-- Logs begin at Tue 2022-10-18 23:51:14 UTC, end at Wed 2022-10-19 00:57:34 UTC. -- Oct 19 00:26:06.284369 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 19 00:26:06.295836 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 19 00:26:06.299067 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Oct 19 00:26:06 2022] *** Oct 19 00:26:06.299265 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 19 00:26:06.299386 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 19 00:26:06.299509 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: nodename: dv-cinder-master-3-iscsi Oct 19 00:26:06.299609 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: machine: x86_64 Oct 19 00:26:06.299704 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: clock source: unix Oct 19 00:26:06.299801 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: pcre jit disabled Oct 19 00:26:06.299927 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: detected number of CPU cores: 4 Oct 19 00:26:06.300040 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: current working directory: / Oct 19 00:26:06.300198 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: detected binary path: /usr/bin/uwsgi-core Oct 19 00:26:06.300366 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: your processes number limit is 47680 Oct 19 00:26:06.300517 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: your memory page size is 4096 bytes Oct 19 00:26:06.300689 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: detected max file descriptor number: 2048 Oct 19 00:26:06.300689 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: lock engine: pthread robust mutexes Oct 19 00:26:06.300951 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: thunder lock: enabled Oct 19 00:26:06.301322 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 19 00:26:06.301322 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 19 00:26:06.350451 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: Python main interpreter initialized at 0x558bdaa6d9c0 Oct 19 00:26:06.350451 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: python threads support enabled Oct 19 00:26:06.350451 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: your server socket listen backlog is limited to 100 connections Oct 19 00:26:06.350451 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: your mercy for graceful operations on workers is 90 seconds Oct 19 00:26:06.350899 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: mapped 403077 bytes (393 KB) for 2 cores Oct 19 00:26:06.350899 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: *** Operational MODE: preforking *** Oct 19 00:26:06.351222 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: *** uWSGI is running in multiple interpreter mode *** Oct 19 00:26:06.351222 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: spawned uWSGI master process (pid: 81798) Oct 19 00:26:06.351490 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 19 00:26:06.352563 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: spawned uWSGI worker 1 (pid: 81811, cores: 1) Oct 19 00:26:06.352563 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: spawned uWSGI worker 2 (pid: 81812, cores: 1) Oct 19 00:26:06.352563 dv-cinder-master-3-iscsi devstack@keystone.service[81798]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 19 00:26:10.729825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 00:26:10.729825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: from cryptography.utils import int_from_bytes Oct 19 00:26:10.730661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 00:26:10.730661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: from cryptography.utils import int_from_bytes Oct 19 00:26:10.734302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 00:26:10.734302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: from cryptography.utils import int_from_bytes Oct 19 00:26:10.734811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 00:26:10.734811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: from cryptography.utils import int_from_bytes Oct 19 00:26:10.781975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 19 00:26:10.782172 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 19 00:26:10.782349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] command line args: [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 19 00:26:10.782661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 19 00:26:10.783571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ================================================================================ {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 19 00:26:10.783694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] admin_token = **** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.783866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] config_dir = [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.784056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.784241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] config_source = [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.784613 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] debug = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.784824 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.785201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.785386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.785644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.785897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.786007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 19 00:26:10.786136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] list_limit = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.786249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 19 00:26:10.786362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] log_config_append = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.786560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] command line args: [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 19 00:26:10.786670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.786670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] log_dir = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.786968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 19 00:26:10.787118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] log_file = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.787220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ================================================================================ {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 19 00:26:10.787346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.787428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] admin_token = **** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.787506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.787632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] config_dir = [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.787709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.787829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.787910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.787991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] config_source = [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] debug = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.788927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789068 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] list_limit = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] log_config_append = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.789912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] log_dir = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.790036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.790114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] log_file = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.790314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.790403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.790591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.790844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.790983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] publish_errors = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791345 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.791896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] standard_threads = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.792903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] use_journal = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] use_json = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] use_stderr = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793676 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] use_syslog = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.793907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.794089 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.794178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.794275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] publish_errors = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.794398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.794551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.794693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.794828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.794911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.794988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.795162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.795245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.795399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.795498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.795652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.795729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] standard_threads = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.795892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.795971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.796145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.796232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.796349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.796433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.796548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] use_journal = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.796628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.796731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] use_json = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.796872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.796995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] use_stderr = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.797074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] use_syslog = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.797145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.797333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 00:26:10.797415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.797584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.797670 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.797839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.797920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.798081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.798161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.798342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.798423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.799196 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.799299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.799567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.799696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.799830 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.799987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.800101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.800252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.800382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.800523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.800639 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.800786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.800894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801284 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801747 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.801987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.802075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.802327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.802544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.802639 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.802822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.802922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.803060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.803141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.803384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.803603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.803691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.803797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.803884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.804055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.804135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.804293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.804375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.804536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.804619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.804778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.804859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.805969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.806139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.806218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.806398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.806509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.806691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.806772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.806928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.807008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.807170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.807323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.807399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.807498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.807653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.807759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.807890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808618 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.808998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.809099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.809262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.809371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.809526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.809632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.809782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.809886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.810985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.811069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.811231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.811349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.811472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.811552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.811715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.811796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.811957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.812982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.813142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.813270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.813381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.813460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.813628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.813715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.813861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.813953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.814150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.814235 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.814398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.814606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.814681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.814882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.814962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.815105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.815193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.815350 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.815432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.815589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.815669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.815829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.815916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.816096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.816176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.816343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.816425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.816597 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.816684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.816842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.816923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.817082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.817164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.817328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.817408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.817569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.817809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.818990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.819090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.819225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.819355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.819510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.819587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.819694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.819825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.819945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.820065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.820186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.820305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.820448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.820573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.820716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.820818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.820968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.821065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.821204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] auth.external = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.821311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.821440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.821552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.821742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.821831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.821980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.822061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.822217 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] auth.password = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.822298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.822452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] auth.token = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.822568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.822719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.822799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.822962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.823042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.823197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.823435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.823525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.823674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.823777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.823918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.824001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.824152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.824239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.824392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] credential.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.824474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.824646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.824769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.824899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.825074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.825197 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.825388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.825468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.825636 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.825717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.825876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.825958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] auth.external = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.826128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.826264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.826387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.826650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.826752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.826910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.826992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.827202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.827283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] auth.password = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.827452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] auth.token = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.827544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.827721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.827801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.827968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.828048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.828231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.828324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.828471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] federation.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.828553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.828712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.828833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.828964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.829117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.829218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.829421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.829502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.829712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] credential.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.829799 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.829995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.830160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.830255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.830517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.830644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.830779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.830881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.831022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.831142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.831272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.831355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.831513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.831672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.831759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.831962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.832066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.832232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.832335 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.832473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.832561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.832726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.832835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.832962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.833138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.833237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.833422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.833504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.833690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.833775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.833932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.834044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.834168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.834272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.834389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.834605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] federation.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.834796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.834877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.834983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.835135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.835219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.835425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.835529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.835701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.835796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.835988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.836085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.836227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.836318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.836473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.836567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.836704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.836810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.836947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.837222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.837407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.837494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.837651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.837747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.837893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.838022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.838292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.838403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.838555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.838686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.838798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.838968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.839063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.839228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.839333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.839514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.839611 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.839772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.839863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.840003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.840087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.840248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.840360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.840486 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.840673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.840764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.840953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.841039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.841194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.841275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.841438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.841526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.841680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.841820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.841932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.842154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.842237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.842401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.842512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.842666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.842824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.842925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.843140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.843222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.843386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.843475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.843625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.843772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.843861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.844055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.844139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.844324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.844410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.844591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.844709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.844825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.844907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.845083 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.845181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.845861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.846036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.846518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.846518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.846688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.846768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.846842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.846916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.846989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.847061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.847198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.847281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.847493 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.847619 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.847750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.847835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.848001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.848085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.848517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.848606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.848769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.848851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.849021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.849133 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.849239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.849315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.849497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.849613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.849741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.849910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.850000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.850198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.850289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.850436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.850605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.850713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.850796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.850969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.851169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.851257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.851460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.851548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.851694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.851775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.851943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.852024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.852181 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.852263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.852428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.852536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.852662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.852793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.852915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.853075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.853156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.853362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.853453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.853616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.853724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.853867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] resource.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.853953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.854121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.854269 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.854367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.854605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.854722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.854869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.854949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.855123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.855257 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.855372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.855585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.855688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.855911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.856075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.856178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.856314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] role.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.856405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.856563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] role.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.856713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.856798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.857018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.857136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.857277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.857358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.857530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.857623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.857784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.857867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.858039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.858154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.858292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.858462 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.858583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.858791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.858904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.859039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.859121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.859291 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.859374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.859543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.859655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.859794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.859900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.860051 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.860218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.860481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.860570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.860719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.860803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.861001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.862845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.863981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.864062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.864237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.864388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.864500 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.864734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.864816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.864987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.865093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] resource.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.865230 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.865465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.865554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.865705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] token.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.865785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.865946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.866027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.866201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.866431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.866532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.866723 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.866858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.866986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.867206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.867344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.867470 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.867596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.867722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.867848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.867996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.868132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.868259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.868385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] role.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.868513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.868639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.868764 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] role.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.868898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.869024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.869149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.869274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.869401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.869535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.869684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.869819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.869946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.870071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.870204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.connection = **** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.870332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.870457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.870645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.870775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.871930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.872189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.872324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.872463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.872598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.872598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.872843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.872969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.872969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.873218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.873358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.873486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.873615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.873744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.873882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.874010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.874134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.874260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.874390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.874390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.874653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.874780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81811) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 19 00:26:10.874907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.875037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.875257 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.875575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.875880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.876173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.876490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.876899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.877347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.877487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.877743 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.878025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.878286 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.878598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] token.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.878894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.879202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.879498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.879818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.880145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.880465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.880777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.881077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.881383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.881663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.881933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.882210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.882542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.882888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.883176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.883494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.883833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.884201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.connection = **** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.884520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.884852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.885180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.885505 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.885811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.886169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.886451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.886770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.887044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.887329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.887644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.887953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.888288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.888616 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.888884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.889165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.889464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.889769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 00:26:10.889976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81812) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 19 00:26:10.904139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.904378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.904597 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.913694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.921386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.921661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.921938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.932439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.942642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.942884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.952843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.953082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:10.998185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.000697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.047479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.047759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.052333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.052600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.205668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.205967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.206134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.206320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.206565 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.206789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.206975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.207154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.207324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.207482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.207701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.207947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.230163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.230388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.230633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.230856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.230995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.231145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.231281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.231414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.231586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.231714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.231843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.231971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.232098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.232347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.243328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.245073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.256510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.256752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.256886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.257035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.257171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.257300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.257427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.257563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.257700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.257827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.257971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.258099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81811) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.258226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.258226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81812) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 00:26:11.275322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.275537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 00:26:11.275669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.275799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 00:26:11.275925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.276071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 00:26:11.276200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.276327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 00:26:11.276478 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.276607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.276743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.276871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.276997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.277125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.277295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.277418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.287372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.287564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.287652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.287890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.288002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.288127 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.288215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.288416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.295263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.295446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.298194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.298346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.298566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.298812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.298900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.299106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.299195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.299396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.299480 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.299710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.299799 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.299977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.300062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.300249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.300348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.300526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.300638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.300802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.300889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.301082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.301186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.301351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.323985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.324112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.325482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.325580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.325766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.325880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.331053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.331164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.331340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.331437 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.331628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.331785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.338013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.338171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.338307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.338461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.342508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.342799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.343086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.344143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.344449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.344741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.345035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.345195 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.345358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.345667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.345863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.345947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.346149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.346268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.346424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.362169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.362286 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.362456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.362573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.366309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.366418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.369518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.369633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.369825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.369912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.373986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.374085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.377706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.377820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.380942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.381053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.381241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.381328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.381528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.381613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.381821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.381908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.382107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.382192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.382392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.382510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.398115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.398243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.398405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.398525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.398705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.398809 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.398975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.399115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.405745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.405889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.407432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.407633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.409651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.409803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.409937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.410066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.412642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.412788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.412918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.413059 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.413187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.413326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.413458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.413587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.413843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.413974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.425131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.425289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.425420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.425548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.425688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.425828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.425959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.426096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.426226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.426467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.426648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.426779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.426911 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.427064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.442842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.443030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.445873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.446010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.448773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.448911 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.450226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.450385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.451715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.451892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.452023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.452164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.452310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.452578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.458973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.459163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.460416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.460650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.463312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.463598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.463730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.463887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.464028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.464173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.464383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.464549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.472756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.473061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.473464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.473622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.473755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.473909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.480524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81812) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.480806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.480939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81811) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 00:26:11.481091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.481221 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.481379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.481543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.481685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.481813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.481939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.482066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.482193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.482338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.482475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.482662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.482791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.482930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.483063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.483191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.483318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.483446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.483596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.483762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.483890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.484019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.484146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81812) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.484274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.484413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81811) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 00:26:11.519866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x558bdaa6d9c0 pid: 81812 (default app) Oct 19 00:26:11.520044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x558bdaa6d9c0 pid: 81811 (default app) Oct 19 00:26:11.523462 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0d4655-22e1-4324-8062-08998479c351 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:11.523749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0d4655-22e1-4324-8062-08998479c351 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:11.524025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0d4655-22e1-4324-8062-08998479c351 None None] PATH_INFO: `/v3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:11.528203 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1/1] 10.222.0.26 () {50 vars in 791 bytes} [Wed Oct 19 00:26:11 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:18.777429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831b570c-9237-4dd2-b377-8a8f865f0cae None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:18.777994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831b570c-9237-4dd2-b377-8a8f865f0cae None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:18.777994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831b570c-9237-4dd2-b377-8a8f865f0cae None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:18.782360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1/2] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:26:18.789013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:18.789598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:18.789598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:18.867679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: INFO dbcounter [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] Registered counter for database keystone Oct 19 00:26:18.889666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG oslo_db.sqlalchemy.engines [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81812) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 19 00:26:18.897474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writer thread running {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 19 00:26:19.109212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG passlib.handlers.bcrypt [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81812) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 19 00:26:19.109687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG passlib.handlers.bcrypt [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81812) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 19 00:26:19.172528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:19.175024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:19.322068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 2/3] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:18 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 534 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 19 00:26:19.798797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831b570c-9237-4dd2-b377-8a8f865f0cae None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:19.874708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: INFO dbcounter [None req-831b570c-9237-4dd2-b377-8a8f865f0cae None None] Registered counter for database keystone Oct 19 00:26:19.898418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG oslo_db.sqlalchemy.engines [None req-831b570c-9237-4dd2-b377-8a8f865f0cae None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81811) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 19 00:26:19.904882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writer thread running {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 19 00:26:20.031408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831b570c-9237-4dd2-b377-8a8f865f0cae None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:20.036433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224ddc2b-4189-4cee-9bae-a119a858928e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:20.037073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224ddc2b-4189-4cee-9bae-a119a858928e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:20.037356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224ddc2b-4189-4cee-9bae-a119a858928e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:20.037634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224ddc2b-4189-4cee-9bae-a119a858928e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:20.046606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224ddc2b-4189-4cee-9bae-a119a858928e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:20.101942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING keystone.common.rbac_enforcer.enforcer [None req-224ddc2b-4189-4cee-9bae-a119a858928e admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 19 00:26:20.102281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224ddc2b-4189-4cee-9bae-a119a858928e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:20.113436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 2/4] 10.222.0.26 () {58 vars in 1324 bytes} [Wed Oct 19 00:26:19 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 00:26:20.119552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:20.137843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73742f1-f449-4859-8062-0ea63686ce7c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:20.143431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4e9f93-0953-4558-95d5-47b95aebcaec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:20.144569 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4e9f93-0953-4558-95d5-47b95aebcaec admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:20.144855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4e9f93-0953-4558-95d5-47b95aebcaec admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:20.145130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4e9f93-0953-4558-95d5-47b95aebcaec admin admin] PATH_INFO: `/v3/projects/d417ad1948a04e309383739147354388` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:20.146777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4e9f93-0953-4558-95d5-47b95aebcaec admin admin] RBAC: Authorizing `identity:get_project(project_id=d417ad1948a04e309383739147354388)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:20.201838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING keystone.common.rbac_enforcer.enforcer [None req-6e4e9f93-0953-4558-95d5-47b95aebcaec admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 19 00:26:20.202185 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4e9f93-0953-4558-95d5-47b95aebcaec admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:20.207994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 3/5] 10.222.0.26 () {56 vars in 1207 bytes} [Wed Oct 19 00:26:20 2022] GET /identity/v3/projects/d417ad1948a04e309383739147354388 => generated 344 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:22.423561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2248ac74-66a7-4d5c-859c-918185e14ace None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:22.424302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2248ac74-66a7-4d5c-859c-918185e14ace None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:22.424302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2248ac74-66a7-4d5c-859c-918185e14ace None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:22.428286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 3/6] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:22.434527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b6bed5-3c92-481d-8ca6-93ced9a1a93d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:22.435341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b6bed5-3c92-481d-8ca6-93ced9a1a93d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:22.435341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b6bed5-3c92-481d-8ca6-93ced9a1a93d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:22.457182 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a6b6bed5-3c92-481d-8ca6-93ced9a1a93d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:22.458827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a6b6bed5-3c92-481d-8ca6-93ced9a1a93d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:22.466946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 4/7] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:22 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 19 00:26:22.734629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2248ac74-66a7-4d5c-859c-918185e14ace None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:22.747785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2248ac74-66a7-4d5c-859c-918185e14ace None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:22.752360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65fbef2-0b96-4b6a-b6b4-b54290fd207c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:22.753018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65fbef2-0b96-4b6a-b6b4-b54290fd207c admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:22.753379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65fbef2-0b96-4b6a-b6b4-b54290fd207c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:22.753697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65fbef2-0b96-4b6a-b6b4-b54290fd207c admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:22.762002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65fbef2-0b96-4b6a-b6b4-b54290fd207c admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:22.763871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65fbef2-0b96-4b6a-b6b4-b54290fd207c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:22.773271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 4/8] 10.222.0.26 () {58 vars in 1324 bytes} [Wed Oct 19 00:26:22 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 00:26:22.779232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b6bed5-3c92-481d-8ca6-93ced9a1a93d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:22.789329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b6bed5-3c92-481d-8ca6-93ced9a1a93d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:22.793782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d018a1-ab5c-4d11-bff9-4216ad692e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:22.794449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d018a1-ab5c-4d11-bff9-4216ad692e5c admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:22.794805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d018a1-ab5c-4d11-bff9-4216ad692e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:22.795117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d018a1-ab5c-4d11-bff9-4216ad692e5c admin admin] PATH_INFO: `/v3/users/ded9b81a3e894fd88539e86bbeb17ee3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:22.796495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d018a1-ab5c-4d11-bff9-4216ad692e5c admin admin] RBAC: Authorizing `identity:get_user(user_id=ded9b81a3e894fd88539e86bbeb17ee3)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:22.799594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d018a1-ab5c-4d11-bff9-4216ad692e5c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:22.801279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 5/9] 10.222.0.26 () {56 vars in 1198 bytes} [Wed Oct 19 00:26:22 2022] GET /identity/v3/users/ded9b81a3e894fd88539e86bbeb17ee3 => generated 247 bytes in 23 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:25.145321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd32325-1307-4bf5-8a71-86a2488ba97e None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:25.146173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd32325-1307-4bf5-8a71-86a2488ba97e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:25.146330 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd32325-1307-4bf5-8a71-86a2488ba97e None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:25.147740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 5/10] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:26:25.153240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8998e515-b8cf-43c9-839a-3da21b606717 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:25.153744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8998e515-b8cf-43c9-839a-3da21b606717 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:25.153960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8998e515-b8cf-43c9-839a-3da21b606717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:25.165947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f157f-f726-41e8-aa51-f23150984c06 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:25.166228 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f157f-f726-41e8-aa51-f23150984c06 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:25.166523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f157f-f726-41e8-aa51-f23150984c06 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:25.167889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 6/11] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:26:25.172979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8998e515-b8cf-43c9-839a-3da21b606717 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:25.173591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:25.173982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:25.174364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:25.174558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8998e515-b8cf-43c9-839a-3da21b606717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:25.217726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG passlib.handlers.bcrypt [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81811) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 19 00:26:25.218120 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG passlib.handlers.bcrypt [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81811) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 19 00:26:25.262248 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 6/12] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:25.268683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:25.270392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:25.379412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 7/13] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:25.531596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8998e515-b8cf-43c9-839a-3da21b606717 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:25.560244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:25.645492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:25.647956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8998e515-b8cf-43c9-839a-3da21b606717 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:25.686327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:25.686987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:25.687330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:25.687610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:25.688305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:25.690093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:25.701082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.sql.core [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 00:26:25.701082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 00:26:25.701082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [parameters: {'id': '07a7b8fa9a8a4641beb6d7895e3bedd1', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 00:26:25.701082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81812) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:26:25.721337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 00:26:25.722328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application yield resource Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise self.value Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application session.commit() Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 00:26:25.723559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.flush() Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application compat.raise_( Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 00:26:25.724648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 00:26:25.725539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:26:25.726507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [parameters: {'id': '07a7b8fa9a8a4641beb6d7895e3bedd1', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 00:26:25.727532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:25.728709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 7/14] 10.222.0.26 () {60 vars in 1137 bytes} [Wed Oct 19 00:26:25 2022] POST /identity/v3/domains => generated 190 bytes in 192 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:25.729759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:25.765836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2837668-0946-4d27-9e86-4c1319547ee1 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:25.768908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:25.812403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2bb97b-f9a9-4b2c-a48e-b4a40a8c555e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:25.814652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bb97b-f9a9-4b2c-a48e-b4a40a8c555e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:25.815119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bb97b-f9a9-4b2c-a48e-b4a40a8c555e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:25.815577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2bb97b-f9a9-4b2c-a48e-b4a40a8c555e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:25.826096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:25.831030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2bb97b-f9a9-4b2c-a48e-b4a40a8c555e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-7f2bb97b-f9a9-4b2c-a48e-b4a40a8c555e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:25.839866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:25.841639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:25.842413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 8/15] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:26:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:26:25.849410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2bb97b-f9a9-4b2c-a48e-b4a40a8c555e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:25.852255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b5a268-8678-480e-b96d-e1b38d485bcf None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:25.888890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:25.889484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:25.889762 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:25.890036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:25.901976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:25.903504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:26:25.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:25.923956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:26:25.924855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:26:25.924855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:26:25.924855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:25.924855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 8/16] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:26:25.928887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:25.962115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2bb97b-f9a9-4b2c-a48e-b4a40a8c555e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:26.003554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:26.004182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:26.004458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:26.004726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:26.005564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:26.005874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:26.008183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:26.026828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 9/17] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:26:26.028326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:26.042615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d4f1b-3e43-4a59-b50b-58357ce01ec2 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:26.067977 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:26.081302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:26.081919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:26.082205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:26.082536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:26.083184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:26.083485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:26.084886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:26.096512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:26.104589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 9/18] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:26:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:26:26.170665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c828f2-731e-4caa-ab48-62d0df4bcb7a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:26.210061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d32ad9-208d-4b6f-ac4a-2c7eac778fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:26.210753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d32ad9-208d-4b6f-ac4a-2c7eac778fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:26.211107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d32ad9-208d-4b6f-ac4a-2c7eac778fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:26.211390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d32ad9-208d-4b6f-ac4a-2c7eac778fb9 None admin] PATH_INFO: `/v3/users/ded9b81a3e894fd88539e86bbeb17ee3` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:26.212988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d32ad9-208d-4b6f-ac4a-2c7eac778fb9 None admin] RBAC: Authorizing `identity:get_user(user_id=ded9b81a3e894fd88539e86bbeb17ee3)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:26.214510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d32ad9-208d-4b6f-ac4a-2c7eac778fb9 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:26.216184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 10/19] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:26 2022] GET /identity/v3/users/ded9b81a3e894fd88539e86bbeb17ee3 => generated 247 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:26:26.223794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:26.342334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b17f594-13c4-4ae7-9a8a-217bd8da6e15 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:26.382278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd04f2-097d-4280-8f2f-db4c0502ed4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:26.382941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd04f2-097d-4280-8f2f-db4c0502ed4b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:26.383215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd04f2-097d-4280-8f2f-db4c0502ed4b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:26.383491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd04f2-097d-4280-8f2f-db4c0502ed4b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:26.384605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acd04f2-097d-4280-8f2f-db4c0502ed4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:26.386070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acd04f2-097d-4280-8f2f-db4c0502ed4b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:26.387926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 10/20] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:26.394262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d32ad9-208d-4b6f-ac4a-2c7eac778fb9 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:26.527190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:26.535846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d32ad9-208d-4b6f-ac4a-2c7eac778fb9 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:26.583278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2815857-b6d5-4ce2-8c5f-e19abb1677d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:26.583980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2815857-b6d5-4ce2-8c5f-e19abb1677d3 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:26.584251 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2815857-b6d5-4ce2-8c5f-e19abb1677d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:26.584530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2815857-b6d5-4ce2-8c5f-e19abb1677d3 None admin] PATH_INFO: `/v3/domains/default/users/ded9b81a3e894fd88539e86bbeb17ee3/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:26.586727 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2815857-b6d5-4ce2-8c5f-e19abb1677d3 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=ded9b81a3e894fd88539e86bbeb17ee3, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:26.588234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2815857-b6d5-4ce2-8c5f-e19abb1677d3 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:26.716717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 11/21] 10.222.0.26 () {58 vars in 1361 bytes} [Wed Oct 19 00:26:26 2022] PUT /identity/v3/domains/default/users/ded9b81a3e894fd88539e86bbeb17ee3/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:26:29.050297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c9456-e76c-4cd1-81bb-2d4c250361f0 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:29.050838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c9456-e76c-4cd1-81bb-2d4c250361f0 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:29.050955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c9456-e76c-4cd1-81bb-2d4c250361f0 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:29.052716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 11/22] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:29.059266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaf10e1-98a4-406a-aa58-002c682a87cd None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:29.059738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaf10e1-98a4-406a-aa58-002c682a87cd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:29.059924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaf10e1-98a4-406a-aa58-002c682a87cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:29.085091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ffaf10e1-98a4-406a-aa58-002c682a87cd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:29.090886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ffaf10e1-98a4-406a-aa58-002c682a87cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:29.196124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 12/23] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:29.355643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fb10d-f11e-4eec-b170-5a842bad3c48 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:29.355963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fb10d-f11e-4eec-b170-5a842bad3c48 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:29.356150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fb10d-f11e-4eec-b170-5a842bad3c48 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:29.357820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 12/24] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:29.363892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412bc653-2dd0-46fc-9e6b-91b9814ef0cd None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:29.364218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412bc653-2dd0-46fc-9e6b-91b9814ef0cd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:29.364530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412bc653-2dd0-46fc-9e6b-91b9814ef0cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:29.393913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-412bc653-2dd0-46fc-9e6b-91b9814ef0cd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:29.395292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-412bc653-2dd0-46fc-9e6b-91b9814ef0cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:29.493343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 13/25] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:29.666706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3278ca-f9e8-430d-8575-8594b78ffdf2 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:29.666996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3278ca-f9e8-430d-8575-8594b78ffdf2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:29.667171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3278ca-f9e8-430d-8575-8594b78ffdf2 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:29.668855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 13/26] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:29.675466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6f5f3c-0aa5-4e62-a2c6-29c72c5ca1ef None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:29.675796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6f5f3c-0aa5-4e62-a2c6-29c72c5ca1ef None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:29.676109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6f5f3c-0aa5-4e62-a2c6-29c72c5ca1ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:29.701253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0c6f5f3c-0aa5-4e62-a2c6-29c72c5ca1ef None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:29.703431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0c6f5f3c-0aa5-4e62-a2c6-29c72c5ca1ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:29.723847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:29.813380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 14/27] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:29.832502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3278ca-f9e8-430d-8575-8594b78ffdf2 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:29.929553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:29.983212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3278ca-f9e8-430d-8575-8594b78ffdf2 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:30.035522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50576e37-671f-4664-9624-b760a17c3010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:30.036289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50576e37-671f-4664-9624-b760a17c3010 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.036582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50576e37-671f-4664-9624-b760a17c3010 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.036897 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50576e37-671f-4664-9624-b760a17c3010 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.057273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50576e37-671f-4664-9624-b760a17c3010 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:30.059209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50576e37-671f-4664-9624-b760a17c3010 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-50576e37-671f-4664-9624-b760a17c3010 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:26:30.079388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:30.080328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:26:30.081149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:26:30.081149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:26:30.081149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:30.081149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 14/28] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:30.088052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6f5f3c-0aa5-4e62-a2c6-29c72c5ca1ef None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:30.137517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55053cfe-d937-4101-a415-9186ba21ba2c None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.137820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55053cfe-d937-4101-a415-9186ba21ba2c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.138073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55053cfe-d937-4101-a415-9186ba21ba2c None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.139721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 15/29] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:30.147131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c37d92-32ab-4919-b9f5-933dad62b6d5 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.147441 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c37d92-32ab-4919-b9f5-933dad62b6d5 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.147673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c37d92-32ab-4919-b9f5-933dad62b6d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.184745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a2c37d92-32ab-4919-b9f5-933dad62b6d5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:30.185877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a2c37d92-32ab-4919-b9f5-933dad62b6d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:30.223662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6f5f3c-0aa5-4e62-a2c6-29c72c5ca1ef None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:30.265063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94696191-a1bd-4b27-b552-9f3123d85117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:30.265705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94696191-a1bd-4b27-b552-9f3123d85117 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.266016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94696191-a1bd-4b27-b552-9f3123d85117 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.266300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94696191-a1bd-4b27-b552-9f3123d85117 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.267056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94696191-a1bd-4b27-b552-9f3123d85117 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:30.267389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94696191-a1bd-4b27-b552-9f3123d85117 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:30.269401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94696191-a1bd-4b27-b552-9f3123d85117 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:30.291549 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 16/30] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:30.292623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c37d92-32ab-4919-b9f5-933dad62b6d5 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:30.296059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 15/31] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:26:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:30.298553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab1aef-9f76-4ab9-8282-e7f1d3bf4769 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.298553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab1aef-9f76-4ab9-8282-e7f1d3bf4769 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.298903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab1aef-9f76-4ab9-8282-e7f1d3bf4769 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.300830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 16/32] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:30.302598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ab1aef-9f76-4ab9-8282-e7f1d3bf4769 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:30.307706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:30.382118 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:30.440523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c37d92-32ab-4919-b9f5-933dad62b6d5 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:30.442015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ab1aef-9f76-4ab9-8282-e7f1d3bf4769 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:30.478931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81496a7d-8064-4b4c-aae7-7c52f89baf5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:30.479721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81496a7d-8064-4b4c-aae7-7c52f89baf5d None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.480102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81496a7d-8064-4b4c-aae7-7c52f89baf5d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.480638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81496a7d-8064-4b4c-aae7-7c52f89baf5d None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.481525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81496a7d-8064-4b4c-aae7-7c52f89baf5d None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:30.483775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81496a7d-8064-4b4c-aae7-7c52f89baf5d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:30.487237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2e9cf5-b93e-4d61-8ead-c44174792f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:30.488145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2e9cf5-b93e-4d61-8ead-c44174792f2c None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.488598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2e9cf5-b93e-4d61-8ead-c44174792f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.489262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2e9cf5-b93e-4d61-8ead-c44174792f2c None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.490176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2e9cf5-b93e-4d61-8ead-c44174792f2c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:30.492737 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2e9cf5-b93e-4d61-8ead-c44174792f2c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:30.537637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 17/33] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:26:30 2022] POST /identity/v3/projects => generated 301 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:26:30.539208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23841d5e-c233-4969-aab5-ea423b1f5fc4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.539577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23841d5e-c233-4969-aab5-ea423b1f5fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.539898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23841d5e-c233-4969-aab5-ea423b1f5fc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.563010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-23841d5e-c233-4969-aab5-ea423b1f5fc4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:30.564497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-23841d5e-c233-4969-aab5-ea423b1f5fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:30.621605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 17/34] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Oct 19 00:26:30 2022] POST /identity/v3/roles => generated 219 bytes in 329 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:30.631319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81496a7d-8064-4b4c-aae7-7c52f89baf5d None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:30.635613 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:30.664717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 18/35] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:30.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23841d5e-c233-4969-aab5-ea423b1f5fc4 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:30.752445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81496a7d-8064-4b4c-aae7-7c52f89baf5d None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:30.763227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:30.794004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4027246f-e3d6-4b1f-ad1d-32459f8fafca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:30.795062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4027246f-e3d6-4b1f-ad1d-32459f8fafca None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.795623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4027246f-e3d6-4b1f-ad1d-32459f8fafca None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.796111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4027246f-e3d6-4b1f-ad1d-32459f8fafca None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4027246f-e3d6-4b1f-ad1d-32459f8fafca None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:30.798972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4027246f-e3d6-4b1f-ad1d-32459f8fafca None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:30.809936 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23841d5e-c233-4969-aab5-ea423b1f5fc4 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:30.827747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 18/36] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Oct 19 00:26:30 2022] POST /identity/v3/roles => generated 225 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:30.873779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b77761b-20c7-49ff-ab03-f000832a718f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:30.874590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b77761b-20c7-49ff-ab03-f000832a718f None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:30.874905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b77761b-20c7-49ff-ab03-f000832a718f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:30.875186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b77761b-20c7-49ff-ab03-f000832a718f None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:30.876593 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b77761b-20c7-49ff-ab03-f000832a718f None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:30.878339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b77761b-20c7-49ff-ab03-f000832a718f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:30.907034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 19/37] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Oct 19 00:26:30 2022] POST /identity/v3/roles => generated 223 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:31.122705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b21fb1-44d6-4d30-b466-ad43c55d680c None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:31.123206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b21fb1-44d6-4d30-b466-ad43c55d680c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:31.123206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b21fb1-44d6-4d30-b466-ad43c55d680c None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:31.124631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 19/38] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:31.131807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb65499-f82b-4a95-8a13-fd16d2671bb8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:31.132347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb65499-f82b-4a95-8a13-fd16d2671bb8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:31.132347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb65499-f82b-4a95-8a13-fd16d2671bb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:31.153602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b21fb1-44d6-4d30-b466-ad43c55d680c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:31.155378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-2fb65499-f82b-4a95-8a13-fd16d2671bb8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:31.156905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2fb65499-f82b-4a95-8a13-fd16d2671bb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:31.241516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:31.270605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 20/39] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:31.302324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b21fb1-44d6-4d30-b466-ad43c55d680c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:31.336782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732a7c1c-9ae1-40a3-950b-8993ac0530e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:31.337422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732a7c1c-9ae1-40a3-950b-8993ac0530e8 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:31.337703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732a7c1c-9ae1-40a3-950b-8993ac0530e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:31.337980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732a7c1c-9ae1-40a3-950b-8993ac0530e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:31.351786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732a7c1c-9ae1-40a3-950b-8993ac0530e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-732a7c1c-9ae1-40a3-950b-8993ac0530e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:31.358098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:31.359160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:31.360145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 20/40] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:26:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:26:31.366011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb65499-f82b-4a95-8a13-fd16d2671bb8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:31.482502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb65499-f82b-4a95-8a13-fd16d2671bb8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:31.521174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:31.521796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:31.522083 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:31.522373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:31.523067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:31.523372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:31.524780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:31.534422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 21/41] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:31.540414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732a7c1c-9ae1-40a3-950b-8993ac0530e8 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:31.560641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:31.577552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:31.613300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:31.673752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732a7c1c-9ae1-40a3-950b-8993ac0530e8 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:31.686240 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a94be9f-5d24-4a66-b531-74a3528bf1bc None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:31.714734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5fa565-bfa1-4a7c-8b26-ef46935bf63f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:31.715874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5fa565-bfa1-4a7c-8b26-ef46935bf63f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:31.716402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5fa565-bfa1-4a7c-8b26-ef46935bf63f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:31.716939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5fa565-bfa1-4a7c-8b26-ef46935bf63f None admin] PATH_INFO: `/v3/users/ded9b81a3e894fd88539e86bbeb17ee3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:31.719162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5fa565-bfa1-4a7c-8b26-ef46935bf63f None admin] RBAC: Authorizing `identity:get_user(user_id=ded9b81a3e894fd88539e86bbeb17ee3)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:31.722681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5fa565-bfa1-4a7c-8b26-ef46935bf63f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:31.730745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 21/42] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:31 2022] GET /identity/v3/users/ded9b81a3e894fd88539e86bbeb17ee3 => generated 247 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:31.730979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea46d68-68bd-4713-9332-b8ba5ec2ee9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:31.730979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea46d68-68bd-4713-9332-b8ba5ec2ee9e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:31.730979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea46d68-68bd-4713-9332-b8ba5ec2ee9e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:31.730979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea46d68-68bd-4713-9332-b8ba5ec2ee9e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:31.731277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea46d68-68bd-4713-9332-b8ba5ec2ee9e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:31.731754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea46d68-68bd-4713-9332-b8ba5ec2ee9e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:31.732165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5fa565-bfa1-4a7c-8b26-ef46935bf63f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:31.733689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 22/43] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:26:31.741306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea46d68-68bd-4713-9332-b8ba5ec2ee9e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:31.860179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:31.863408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea46d68-68bd-4713-9332-b8ba5ec2ee9e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:31.868974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5fa565-bfa1-4a7c-8b26-ef46935bf63f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:31.872321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:31.905799 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58104f47-ccf5-49d6-b335-9b2d98885c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:31.906473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58104f47-ccf5-49d6-b335-9b2d98885c82 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:31.906790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58104f47-ccf5-49d6-b335-9b2d98885c82 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:31.907107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58104f47-ccf5-49d6-b335-9b2d98885c82 None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:31.907999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58104f47-ccf5-49d6-b335-9b2d98885c82 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:31.909592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58104f47-ccf5-49d6-b335-9b2d98885c82 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:31.913954 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26564f97-5abb-44d4-96d7-83de7713d62f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:31.914779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26564f97-5abb-44d4-96d7-83de7713d62f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:31.915065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26564f97-5abb-44d4-96d7-83de7713d62f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:31.915677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26564f97-5abb-44d4-96d7-83de7713d62f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:31.916921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26564f97-5abb-44d4-96d7-83de7713d62f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:31.918398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26564f97-5abb-44d4-96d7-83de7713d62f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:31.920435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 22/44] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:31.927314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26564f97-5abb-44d4-96d7-83de7713d62f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:31.946868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 23/45] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:26:31 2022] POST /identity/v3/projects => generated 312 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:26:32.051369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26564f97-5abb-44d4-96d7-83de7713d62f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:32.091808 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc2b862-af5f-46ac-b74f-7324d53a09eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:32.092444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc2b862-af5f-46ac-b74f-7324d53a09eb None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:32.092756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc2b862-af5f-46ac-b74f-7324d53a09eb None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:32.093068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc2b862-af5f-46ac-b74f-7324d53a09eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:32.093860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc2b862-af5f-46ac-b74f-7324d53a09eb None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.domain.id=default, user.id=ded9b81a3e894fd88539e86bbeb17ee3) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:32.094207 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc2b862-af5f-46ac-b74f-7324d53a09eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:32.096619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc2b862-af5f-46ac-b74f-7324d53a09eb None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:32.110715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 23/46] 10.222.0.26 () {56 vars in 1322 bytes} [Wed Oct 19 00:26:31 2022] GET /identity/v3/role_assignments?user.id=ded9b81a3e894fd88539e86bbeb17ee3&scope.domain.id=default&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 530 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:26:34.127369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e8c319-c4ec-4d9c-b26b-7a44563717a7 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:34.127909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e8c319-c4ec-4d9c-b26b-7a44563717a7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:34.127909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e8c319-c4ec-4d9c-b26b-7a44563717a7 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:34.129312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 24/47] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:34.136078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075656a2-60e1-46d0-8979-fa4ebef015c1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:34.136574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075656a2-60e1-46d0-8979-fa4ebef015c1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:34.136655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075656a2-60e1-46d0-8979-fa4ebef015c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:34.156763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-075656a2-60e1-46d0-8979-fa4ebef015c1 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:34.158372 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-075656a2-60e1-46d0-8979-fa4ebef015c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:34.182642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:34.336929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 24/48] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:34.608379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e8c319-c4ec-4d9c-b26b-7a44563717a7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:34.711652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e8c319-c4ec-4d9c-b26b-7a44563717a7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:34.745867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e371a61-903a-42a5-badb-a0be3ce1f4d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:34.746517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e371a61-903a-42a5-badb-a0be3ce1f4d3 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:34.746805 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e371a61-903a-42a5-badb-a0be3ce1f4d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:34.747078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e371a61-903a-42a5-badb-a0be3ce1f4d3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:34.748234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e371a61-903a-42a5-badb-a0be3ce1f4d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:34.749687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e371a61-903a-42a5-badb-a0be3ce1f4d3 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:34.751295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 25/49] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:34.759377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075656a2-60e1-46d0-8979-fa4ebef015c1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:34.860091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075656a2-60e1-46d0-8979-fa4ebef015c1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:34.894025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861dea71-e5e9-435e-a5f4-e6a8ee4e5d86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:34.894670 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861dea71-e5e9-435e-a5f4-e6a8ee4e5d86 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:34.894955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861dea71-e5e9-435e-a5f4-e6a8ee4e5d86 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:34.895230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861dea71-e5e9-435e-a5f4-e6a8ee4e5d86 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:34.895907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861dea71-e5e9-435e-a5f4-e6a8ee4e5d86 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:34.897430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861dea71-e5e9-435e-a5f4-e6a8ee4e5d86 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:34.926312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 25/50] 10.222.0.26 () {60 vars in 1140 bytes} [Wed Oct 19 00:26:34 2022] POST /identity/v3/projects => generated 298 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:37.136002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92f539a-4423-4883-bdc1-b001ca6c571d None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:37.136579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92f539a-4423-4883-bdc1-b001ca6c571d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:37.136579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92f539a-4423-4883-bdc1-b001ca6c571d None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:37.137915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 26/51] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:37.144095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48f895-50df-4dbb-873e-afc128c8688d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:37.144713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48f895-50df-4dbb-873e-afc128c8688d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:37.144713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48f895-50df-4dbb-873e-afc128c8688d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:37.163330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6e48f895-50df-4dbb-873e-afc128c8688d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:37.164603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6e48f895-50df-4dbb-873e-afc128c8688d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:37.168017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:37.236462 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 26/52] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:37.507537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92f539a-4423-4883-bdc1-b001ca6c571d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:37.535927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:37.722411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92f539a-4423-4883-bdc1-b001ca6c571d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:37.758056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e678394-0895-445a-b208-e11e7ee79aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:37.758645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e678394-0895-445a-b208-e11e7ee79aa3 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:37.758936 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e678394-0895-445a-b208-e11e7ee79aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:37.759218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e678394-0895-445a-b208-e11e7ee79aa3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:37.760350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e678394-0895-445a-b208-e11e7ee79aa3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:37.761803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e678394-0895-445a-b208-e11e7ee79aa3 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:37.763429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 27/53] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:37.769984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e48f895-50df-4dbb-873e-afc128c8688d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:37.874907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e48f895-50df-4dbb-873e-afc128c8688d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:37.908961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d014f8ac-ee83-4daa-a955-3fc5cf4b943a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:37.909573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014f8ac-ee83-4daa-a955-3fc5cf4b943a None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:37.909858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014f8ac-ee83-4daa-a955-3fc5cf4b943a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:37.910134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014f8ac-ee83-4daa-a955-3fc5cf4b943a None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:37.910842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d014f8ac-ee83-4daa-a955-3fc5cf4b943a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:37.912273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d014f8ac-ee83-4daa-a955-3fc5cf4b943a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:37.929291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:37.944217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 27/54] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:26:37 2022] POST /identity/v3/users => generated 275 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:41.315275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ea6c1f-6252-4a6d-96ea-e9c12a217dfa None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:41.316905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ea6c1f-6252-4a6d-96ea-e9c12a217dfa None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:41.317723 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ea6c1f-6252-4a6d-96ea-e9c12a217dfa None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:41.320758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 28/55] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:41 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:41.327776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b2a71d-fe23-4420-95ad-efc3bcb798ba None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:41.330734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b2a71d-fe23-4420-95ad-efc3bcb798ba None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:41.331548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b2a71d-fe23-4420-95ad-efc3bcb798ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:41.379600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-21b2a71d-fe23-4420-95ad-efc3bcb798ba None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:41.382144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-21b2a71d-fe23-4420-95ad-efc3bcb798ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:41.459385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8c054-fcc0-4e2f-9bd8-e733e8fd73b9 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:41.460321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8c054-fcc0-4e2f-9bd8-e733e8fd73b9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:41.461085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8c054-fcc0-4e2f-9bd8-e733e8fd73b9 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:41.463903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 29/56] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:41 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:41.471332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f672d77b-f2ef-4de1-ac28-02279a38d4c6 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:41.472451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f672d77b-f2ef-4de1-ac28-02279a38d4c6 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:41.473699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f672d77b-f2ef-4de1-ac28-02279a38d4c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:41.511522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 28/57] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:41.530966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f672d77b-f2ef-4de1-ac28-02279a38d4c6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:41.532543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f672d77b-f2ef-4de1-ac28-02279a38d4c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:41.635936 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 30/58] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:41.794188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b2a71d-fe23-4420-95ad-efc3bcb798ba None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:41.942546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b2a71d-fe23-4420-95ad-efc3bcb798ba None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:41.970988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ab6e1-6254-428f-8724-f4fb76474a0a None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:41.971690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ab6e1-6254-428f-8724-f4fb76474a0a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:41.972153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ab6e1-6254-428f-8724-f4fb76474a0a None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:41.973976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 31/59] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:41.975870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:41.984430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc68508e-f367-4089-85b5-c88b2825ead0 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:41.984717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc68508e-f367-4089-85b5-c88b2825ead0 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:41.985051 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc68508e-f367-4089-85b5-c88b2825ead0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:41.995635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1763f8-404c-4ee6-bdc4-13b61636ae34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:41.996295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1763f8-404c-4ee6-bdc4-13b61636ae34 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.002088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1763f8-404c-4ee6-bdc4-13b61636ae34 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.002088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1763f8-404c-4ee6-bdc4-13b61636ae34 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.011692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-fc68508e-f367-4089-85b5-c88b2825ead0 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:42.012913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fc68508e-f367-4089-85b5-c88b2825ead0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:42.015650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1763f8-404c-4ee6-bdc4-13b61636ae34 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-3a1763f8-404c-4ee6-bdc4-13b61636ae34 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:42.029145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:42.030094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:42.030896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:42.031463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 29/60] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:42.036125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228e5bb2-0aad-4c49-8096-f3c30a4d1628 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.037004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228e5bb2-0aad-4c49-8096-f3c30a4d1628 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.037709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228e5bb2-0aad-4c49-8096-f3c30a4d1628 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.039667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:42.041377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 30/61] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:42 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:42.042997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228e5bb2-0aad-4c49-8096-f3c30a4d1628 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.111374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 32/62] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:42.113025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f190bd-dbb9-4802-a737-9a023ab91bc3 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.113461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f190bd-dbb9-4802-a737-9a023ab91bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.113915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f190bd-dbb9-4802-a737-9a023ab91bc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.137362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-29f190bd-dbb9-4802-a737-9a023ab91bc3 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:42.138826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-29f190bd-dbb9-4802-a737-9a023ab91bc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:42.190768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228e5bb2-0aad-4c49-8096-f3c30a4d1628 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:42.229784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:42.230676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.231098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.231449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 33/63] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:42.238507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.238696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f190bd-dbb9-4802-a737-9a023ab91bc3 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.239736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] RBAC: Adding query filter params (name=member) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:42.240226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:42.241925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:42.264771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 31/64] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:42.270988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.396110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:42.402652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:42.420341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f190bd-dbb9-4802-a737-9a023ab91bc3 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:42.457619 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc570f-62c2-412a-bda3-68a281abe9d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:42.458268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc570f-62c2-412a-bda3-68a281abe9d3 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.458589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc570f-62c2-412a-bda3-68a281abe9d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.458864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc570f-62c2-412a-bda3-68a281abe9d3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.469415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc570f-62c2-412a-bda3-68a281abe9d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-6bdc570f-62c2-412a-bda3-68a281abe9d3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:42.474855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:42.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:42.476479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 34/65] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:42.478067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc570f-62c2-412a-bda3-68a281abe9d3 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.490701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376d2f5c-ff03-4e09-818f-72bee53739ff None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:42.525925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6b38fc-3c52-4e4b-ba22-d9fcde47a993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:42.526611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6b38fc-3c52-4e4b-ba22-d9fcde47a993 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.526896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6b38fc-3c52-4e4b-ba22-d9fcde47a993 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.527178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6b38fc-3c52-4e4b-ba22-d9fcde47a993 None admin] PATH_INFO: `/v3/users/a43f8e46711349d4bcf7bab7f618f326` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.542790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6b38fc-3c52-4e4b-ba22-d9fcde47a993 None admin] RBAC: Authorizing `identity:get_user(user_id=a43f8e46711349d4bcf7bab7f618f326)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:42.544512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6b38fc-3c52-4e4b-ba22-d9fcde47a993 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:42.546306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 32/66] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/users/a43f8e46711349d4bcf7bab7f618f326 => generated 275 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:42.548501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eb5132-81ea-4d4b-ac31-28fe7268a6a6 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.548874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eb5132-81ea-4d4b-ac31-28fe7268a6a6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.549341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eb5132-81ea-4d4b-ac31-28fe7268a6a6 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.550602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 33/67] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:42.551433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eb5132-81ea-4d4b-ac31-28fe7268a6a6 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.604106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc570f-62c2-412a-bda3-68a281abe9d3 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:42.638624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee7719d-f7ee-420b-84a1-60335225a4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:42.639279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee7719d-f7ee-420b-84a1-60335225a4d1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.639550 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee7719d-f7ee-420b-84a1-60335225a4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.639832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee7719d-f7ee-420b-84a1-60335225a4d1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.651997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee7719d-f7ee-420b-84a1-60335225a4d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-0ee7719d-f7ee-420b-84a1-60335225a4d1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:42.658018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:42.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:42.661165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 35/68] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:42.662400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668af2a5-78d4-40f6-91a8-b6d8b0b6c31f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.662400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668af2a5-78d4-40f6-91a8-b6d8b0b6c31f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.663380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668af2a5-78d4-40f6-91a8-b6d8b0b6c31f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.668779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eb5132-81ea-4d4b-ac31-28fe7268a6a6 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:42.671894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:42.677881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:42.683571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-668af2a5-78d4-40f6-91a8-b6d8b0b6c31f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:42.684798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-668af2a5-78d4-40f6-91a8-b6d8b0b6c31f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:42.707071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:42.707691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.707977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.708259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.708937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:42.709245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:42.710682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:42.718835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 34/69] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:42.719980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.755315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 36/70] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:42.755871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668af2a5-78d4-40f6-91a8-b6d8b0b6c31f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.853816 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d405032-8925-4df6-a18a-4244e6eca929 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:42.867013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668af2a5-78d4-40f6-91a8-b6d8b0b6c31f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:42.893145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676a302a-5306-4cfe-938a-8eb886f09f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:42.894029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a302a-5306-4cfe-938a-8eb886f09f68 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.894371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a302a-5306-4cfe-938a-8eb886f09f68 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.894565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a302a-5306-4cfe-938a-8eb886f09f68 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.895875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676a302a-5306-4cfe-938a-8eb886f09f68 None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:42.897697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676a302a-5306-4cfe-938a-8eb886f09f68 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:42.900196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 35/71] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:42.900196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676a302a-5306-4cfe-938a-8eb886f09f68 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.910560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9e94b-1818-4f1b-93a3-d81dac2f30a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:42.911201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9e94b-1818-4f1b-93a3-d81dac2f30a2 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:42.911533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9e94b-1818-4f1b-93a3-d81dac2f30a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:42.912287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9e94b-1818-4f1b-93a3-d81dac2f30a2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:42.915688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:42.929933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f9e94b-1818-4f1b-93a3-d81dac2f30a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-24f9e94b-1818-4f1b-93a3-d81dac2f30a2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:42.934934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:42.935790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 37/72] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:42.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9e94b-1818-4f1b-93a3-d81dac2f30a2 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:42.961956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:43.027772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676a302a-5306-4cfe-938a-8eb886f09f68 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.051765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9e94b-1818-4f1b-93a3-d81dac2f30a2 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.070356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.071037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.071323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.071607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.072279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:43.072590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.073989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.082762 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 36/73] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:43.083362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.094790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ea4f74-d1df-4d51-9d66-dd2711a60697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.095440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ea4f74-d1df-4d51-9d66-dd2711a60697 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.095729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ea4f74-d1df-4d51-9d66-dd2711a60697 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.096014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ea4f74-d1df-4d51-9d66-dd2711a60697 None admin] PATH_INFO: `/v3/users/a43f8e46711349d4bcf7bab7f618f326` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.097194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ea4f74-d1df-4d51-9d66-dd2711a60697 None admin] RBAC: Authorizing `identity:get_user(user_id=a43f8e46711349d4bcf7bab7f618f326)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.098963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ea4f74-d1df-4d51-9d66-dd2711a60697 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.100571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 38/74] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:42 2022] GET /identity/v3/users/a43f8e46711349d4bcf7bab7f618f326 => generated 275 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:43.101344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ea4f74-d1df-4d51-9d66-dd2711a60697 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.168403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:43.187667 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:43.192315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f517b32c-5200-48c4-94e4-54d9877f3d25 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.210036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ea4f74-d1df-4d51-9d66-dd2711a60697 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.227547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de74048-c57b-4a2b-89f4-51397c6d9b18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.228221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de74048-c57b-4a2b-89f4-51397c6d9b18 None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.228507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de74048-c57b-4a2b-89f4-51397c6d9b18 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.228787 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de74048-c57b-4a2b-89f4-51397c6d9b18 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/users/a43f8e46711349d4bcf7bab7f618f326/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.230846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de74048-c57b-4a2b-89f4-51397c6d9b18 None admin] RBAC: Authorizing `identity:create_grant(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362, user_id=a43f8e46711349d4bcf7bab7f618f326, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.232320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de74048-c57b-4a2b-89f4-51397c6d9b18 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.245649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.246333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.246653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.246940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.247056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 37/75] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:26:43 2022] PUT /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/users/a43f8e46711349d4bcf7bab7f618f326/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:26:43.247612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:43.247920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.248028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de74048-c57b-4a2b-89f4-51397c6d9b18 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.249518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.271876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 39/76] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:26:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:43.271876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.387337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de74048-c57b-4a2b-89f4-51397c6d9b18 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.395351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033b7210-3577-4e0b-b213-e8f5696b7ae9 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.423856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:43.430522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f56d0-f00b-47a8-966f-dc2769a51088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.431032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f56d0-f00b-47a8-966f-dc2769a51088 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.431032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f56d0-f00b-47a8-966f-dc2769a51088 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.431162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f56d0-f00b-47a8-966f-dc2769a51088 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.432682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824f56d0-f00b-47a8-966f-dc2769a51088 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.433799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824f56d0-f00b-47a8-966f-dc2769a51088 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.435773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 38/77] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:43.436218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f56d0-f00b-47a8-966f-dc2769a51088 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.439102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52958b32-bd75-43d3-a78d-7cdb479830cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.439767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52958b32-bd75-43d3-a78d-7cdb479830cd None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.440056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52958b32-bd75-43d3-a78d-7cdb479830cd None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.440369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52958b32-bd75-43d3-a78d-7cdb479830cd None admin] PATH_INFO: `/v3/users/ded9b81a3e894fd88539e86bbeb17ee3` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.442552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52958b32-bd75-43d3-a78d-7cdb479830cd None admin] RBAC: Authorizing `identity:get_user(user_id=ded9b81a3e894fd88539e86bbeb17ee3)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.443137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52958b32-bd75-43d3-a78d-7cdb479830cd None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.445307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 40/78] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:43 2022] GET /identity/v3/users/ded9b81a3e894fd88539e86bbeb17ee3 => generated 247 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:43.445584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52958b32-bd75-43d3-a78d-7cdb479830cd None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.472710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:43.555942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f56d0-f00b-47a8-966f-dc2769a51088 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.564590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52958b32-bd75-43d3-a78d-7cdb479830cd None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.593266 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0a0a03-5022-4e59-bafe-fd56794898d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.593906 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0a0a03-5022-4e59-bafe-fd56794898d6 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.595006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0a0a03-5022-4e59-bafe-fd56794898d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.595112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0a0a03-5022-4e59-bafe-fd56794898d6 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.596227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0a0a03-5022-4e59-bafe-fd56794898d6 None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.597681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0a0a03-5022-4e59-bafe-fd56794898d6 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.599619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 39/79] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:43 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:43.600361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0a0a03-5022-4e59-bafe-fd56794898d6 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.608070 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7db8f9-7a87-4e53-9d56-77db5c88465c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.608727 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7db8f9-7a87-4e53-9d56-77db5c88465c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.609016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7db8f9-7a87-4e53-9d56-77db5c88465c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.609302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7db8f9-7a87-4e53-9d56-77db5c88465c None admin] PATH_INFO: `/v3/users/a43f8e46711349d4bcf7bab7f618f326` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.610724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7db8f9-7a87-4e53-9d56-77db5c88465c None admin] RBAC: Authorizing `identity:get_user(user_id=a43f8e46711349d4bcf7bab7f618f326)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.612192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7db8f9-7a87-4e53-9d56-77db5c88465c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.613936 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 41/80] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:43 2022] GET /identity/v3/users/a43f8e46711349d4bcf7bab7f618f326 => generated 275 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:43.614591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7db8f9-7a87-4e53-9d56-77db5c88465c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.682077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:43.712460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:43.722007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0a0a03-5022-4e59-bafe-fd56794898d6 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.733930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7db8f9-7a87-4e53-9d56-77db5c88465c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.759109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da42bfc6-604a-4536-8759-753e7722e2ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.759738 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da42bfc6-604a-4536-8759-753e7722e2ef None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.760025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da42bfc6-604a-4536-8759-753e7722e2ef None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.760303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da42bfc6-604a-4536-8759-753e7722e2ef None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.760978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da42bfc6-604a-4536-8759-753e7722e2ef None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.762472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da42bfc6-604a-4536-8759-753e7722e2ef None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.776157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4978fb28-aeb8-470a-806e-0c39bca894a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.777160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4978fb28-aeb8-470a-806e-0c39bca894a0 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.777557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4978fb28-aeb8-470a-806e-0c39bca894a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.777932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4978fb28-aeb8-470a-806e-0c39bca894a0 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.779422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4978fb28-aeb8-470a-806e-0c39bca894a0 None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4978fb28-aeb8-470a-806e-0c39bca894a0 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:43.784076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 42/81] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:43 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:43.784860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4978fb28-aeb8-470a-806e-0c39bca894a0 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.792828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 40/82] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:26:43 2022] POST /identity/v3/users => generated 289 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:43.794130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da42bfc6-604a-4536-8759-753e7722e2ef None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:43.929765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4978fb28-aeb8-470a-806e-0c39bca894a0 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.963758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:43.973354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da42bfc6-604a-4536-8759-753e7722e2ef None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:43.978851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a6b864-ee54-4347-bf86-2ae34480d660 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:43.979896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a6b864-ee54-4347-bf86-2ae34480d660 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:43.980421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a6b864-ee54-4347-bf86-2ae34480d660 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:43.980898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a6b864-ee54-4347-bf86-2ae34480d660 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/users/a43f8e46711349d4bcf7bab7f618f326/roles/66029780d52042b9a44109eb91c0d80e` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:43.984035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a6b864-ee54-4347-bf86-2ae34480d660 None admin] RBAC: Authorizing `identity:create_grant(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362, user_id=a43f8e46711349d4bcf7bab7f618f326, role_id=66029780d52042b9a44109eb91c0d80e)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:43.986031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a6b864-ee54-4347-bf86-2ae34480d660 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:44.007720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 43/83] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:26:43 2022] PUT /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/users/a43f8e46711349d4bcf7bab7f618f326/roles/66029780d52042b9a44109eb91c0d80e => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:26:44.009031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a6b864-ee54-4347-bf86-2ae34480d660 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:44.024816 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e175e29-8ba0-4740-bac9-8d918de1fa9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:44.025902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e175e29-8ba0-4740-bac9-8d918de1fa9c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:44.026433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e175e29-8ba0-4740-bac9-8d918de1fa9c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:44.027017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e175e29-8ba0-4740-bac9-8d918de1fa9c None admin] PATH_INFO: `/v3/projects/d3942a754a534cdc8791e7c3faec8aaa` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:44.028574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e175e29-8ba0-4740-bac9-8d918de1fa9c None admin] RBAC: Authorizing `identity:get_project(project_id=d3942a754a534cdc8791e7c3faec8aaa)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:44.030547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e175e29-8ba0-4740-bac9-8d918de1fa9c None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:44.033502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 41/84] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:43 2022] GET /identity/v3/projects/d3942a754a534cdc8791e7c3faec8aaa => generated 312 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:44.040142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e175e29-8ba0-4740-bac9-8d918de1fa9c None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:44.074652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:44.148555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a6b864-ee54-4347-bf86-2ae34480d660 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:44.197582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e175e29-8ba0-4740-bac9-8d918de1fa9c None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:44.199888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462faed2-fa0f-440a-a8c3-c7b683fa804f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:44.201898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462faed2-fa0f-440a-a8c3-c7b683fa804f None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:44.202549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462faed2-fa0f-440a-a8c3-c7b683fa804f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:44.203073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462faed2-fa0f-440a-a8c3-c7b683fa804f None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/users/ded9b81a3e894fd88539e86bbeb17ee3/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:44.205702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462faed2-fa0f-440a-a8c3-c7b683fa804f None admin] RBAC: Authorizing `identity:create_grant(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362, user_id=ded9b81a3e894fd88539e86bbeb17ee3, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:44.207690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462faed2-fa0f-440a-a8c3-c7b683fa804f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:44.228015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 44/85] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:26:44 2022] PUT /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/users/ded9b81a3e894fd88539e86bbeb17ee3/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:26:44.253989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4d921-ec12-4ab1-831b-29251a9e0a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:44.254822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4d921-ec12-4ab1-831b-29251a9e0a5a None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:44.255106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4d921-ec12-4ab1-831b-29251a9e0a5a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:44.255382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4d921-ec12-4ab1-831b-29251a9e0a5a None admin] PATH_INFO: `/v3/projects/d3942a754a534cdc8791e7c3faec8aaa/users/a43f8e46711349d4bcf7bab7f618f326/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:44.257536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4d921-ec12-4ab1-831b-29251a9e0a5a None admin] RBAC: Authorizing `identity:create_grant(project_id=d3942a754a534cdc8791e7c3faec8aaa, user_id=a43f8e46711349d4bcf7bab7f618f326, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:44.259191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4d921-ec12-4ab1-831b-29251a9e0a5a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:44.273770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 42/86] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:26:44 2022] PUT /identity/v3/projects/d3942a754a534cdc8791e7c3faec8aaa/users/a43f8e46711349d4bcf7bab7f618f326/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:26:45.814232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025c5f0f-d195-4423-b391-f3823adc1298 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:45.815669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025c5f0f-d195-4423-b391-f3823adc1298 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:45.816413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025c5f0f-d195-4423-b391-f3823adc1298 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:45.819087 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 45/87] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:45 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:45.825684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e01fff-48d5-4c61-a4d8-ff687c11560a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:45.827091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e01fff-48d5-4c61-a4d8-ff687c11560a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:45.827852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e01fff-48d5-4c61-a4d8-ff687c11560a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:45.864208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-66e01fff-48d5-4c61-a4d8-ff687c11560a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:45.866624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-66e01fff-48d5-4c61-a4d8-ff687c11560a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:45.970905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 43/88] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:46.515770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025c5f0f-d195-4423-b391-f3823adc1298 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:46.576667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:46.648814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025c5f0f-d195-4423-b391-f3823adc1298 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:46.692756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa1e43-d7e9-4000-a06e-3d832e360378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:46.693416 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa1e43-d7e9-4000-a06e-3d832e360378 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:46.693698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa1e43-d7e9-4000-a06e-3d832e360378 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:46.693966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa1e43-d7e9-4000-a06e-3d832e360378 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:46.706541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aa1e43-d7e9-4000-a06e-3d832e360378 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-74aa1e43-d7e9-4000-a06e-3d832e360378 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:46.716025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:46.717788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:46.721577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 46/89] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:46.724432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e01fff-48d5-4c61-a4d8-ff687c11560a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:46.734630 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:46.851053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e01fff-48d5-4c61-a4d8-ff687c11560a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:46.886209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:46.886875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:46.890882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:46.891710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:46.892420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:46.892736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:46.894152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:46.903275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 44/90] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:26:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:46.909482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa1e43-d7e9-4000-a06e-3d832e360378 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:47.079646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa1e43-d7e9-4000-a06e-3d832e360378 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:47.096384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:47.137296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438464fe-39c6-47af-8865-c8583108b07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:47.137868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438464fe-39c6-47af-8865-c8583108b07e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.138157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438464fe-39c6-47af-8865-c8583108b07e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.138440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438464fe-39c6-47af-8865-c8583108b07e None admin] PATH_INFO: `/v3/users/a43f8e46711349d4bcf7bab7f618f326` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.139688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438464fe-39c6-47af-8865-c8583108b07e None admin] RBAC: Authorizing `identity:get_user(user_id=a43f8e46711349d4bcf7bab7f618f326)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:47.141884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438464fe-39c6-47af-8865-c8583108b07e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:47.144062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 47/91] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:46 2022] GET /identity/v3/users/a43f8e46711349d4bcf7bab7f618f326 => generated 275 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:47.150009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:47.261335 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:47.265852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d932797-af81-40db-9668-03bed1488083 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.266852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d932797-af81-40db-9668-03bed1488083 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.267144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d932797-af81-40db-9668-03bed1488083 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.268710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 48/92] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:26:47.276525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ccc0d5-8c19-43d3-84b4-fed5999d151e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.276872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ccc0d5-8c19-43d3-84b4-fed5999d151e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.277102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ccc0d5-8c19-43d3-84b4-fed5999d151e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.310133 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3099c0c-f140-423f-b43d-7b7719bf36c6 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:47.323405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f4ccc0d5-8c19-43d3-84b4-fed5999d151e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:47.324767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f4ccc0d5-8c19-43d3-84b4-fed5999d151e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:47.358882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185319be-94cc-477a-8fef-5a9b41afa528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:47.359534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185319be-94cc-477a-8fef-5a9b41afa528 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.361041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185319be-94cc-477a-8fef-5a9b41afa528 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.361345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185319be-94cc-477a-8fef-5a9b41afa528 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.447447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185319be-94cc-477a-8fef-5a9b41afa528 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:47.448949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185319be-94cc-477a-8fef-5a9b41afa528 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:47.503370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 49/93] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:47.532381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:47.561237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f2be8-f5c2-4520-86aa-e6c7f2e12363 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.561237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f2be8-f5c2-4520-86aa-e6c7f2e12363 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.561237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f2be8-f5c2-4520-86aa-e6c7f2e12363 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.561237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 50/94] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:47.563254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688abb3-a1c3-45ec-9db0-7c031a95a2bd None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.563494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688abb3-a1c3-45ec-9db0-7c031a95a2bd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.570517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688abb3-a1c3-45ec-9db0-7c031a95a2bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.581874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 45/95] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:26:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:26:47.590011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185319be-94cc-477a-8fef-5a9b41afa528 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:47.619743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-4688abb3-a1c3-45ec-9db0-7c031a95a2bd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:47.620989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4688abb3-a1c3-45ec-9db0-7c031a95a2bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:47.767630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 51/96] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:47.768879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182f149-328d-479b-85f6-89dba0663a52 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.769169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182f149-328d-479b-85f6-89dba0663a52 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.769447 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9182f149-328d-479b-85f6-89dba0663a52 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.776508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 52/97] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:47 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:47.783521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b368b70-f7b1-4fbd-9ee9-451fd2bf4882 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.783846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b368b70-f7b1-4fbd-9ee9-451fd2bf4882 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.784077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b368b70-f7b1-4fbd-9ee9-451fd2bf4882 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.789818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185319be-94cc-477a-8fef-5a9b41afa528 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:47.820713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5b368b70-f7b1-4fbd-9ee9-451fd2bf4882 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:47.821926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5b368b70-f7b1-4fbd-9ee9-451fd2bf4882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:47.870008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38db1f5-c750-45f4-a1e8-c5344761d5fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:47.870700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38db1f5-c750-45f4-a1e8-c5344761d5fd None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:47.870989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38db1f5-c750-45f4-a1e8-c5344761d5fd None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:47.871268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38db1f5-c750-45f4-a1e8-c5344761d5fd None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:47.873565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38db1f5-c750-45f4-a1e8-c5344761d5fd None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:47.877219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38db1f5-c750-45f4-a1e8-c5344761d5fd None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:47.884683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 46/98] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:47 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:47.891850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38db1f5-c750-45f4-a1e8-c5344761d5fd None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:47.914784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:47.977120 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 53/99] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:47.980843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b368b70-f7b1-4fbd-9ee9-451fd2bf4882 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:47.991360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:48.149730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38db1f5-c750-45f4-a1e8-c5344761d5fd None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:48.150025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b368b70-f7b1-4fbd-9ee9-451fd2bf4882 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:48.195717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030daacf-a44d-4be1-bdd5-b94bdbc5631e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:48.196390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030daacf-a44d-4be1-bdd5-b94bdbc5631e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.196679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030daacf-a44d-4be1-bdd5-b94bdbc5631e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.196960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030daacf-a44d-4be1-bdd5-b94bdbc5631e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.213794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030daacf-a44d-4be1-bdd5-b94bdbc5631e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-030daacf-a44d-4be1-bdd5-b94bdbc5631e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:48.218927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:48.219877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:26:48.220642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:48.221317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 54/100] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:26:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:48.224057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d5859-f3ea-4a42-8f96-9184f657d6f8 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.224400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d5859-f3ea-4a42-8f96-9184f657d6f8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.224690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d5859-f3ea-4a42-8f96-9184f657d6f8 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.226290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 55/101] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:48.231740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d5859-f3ea-4a42-8f96-9184f657d6f8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:48.259290 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b5140d-8e1e-458a-9589-7c9cc05b3ed5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:48.259926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b5140d-8e1e-458a-9589-7c9cc05b3ed5 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.260212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b5140d-8e1e-458a-9589-7c9cc05b3ed5 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.260561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b5140d-8e1e-458a-9589-7c9cc05b3ed5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.261312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b5140d-8e1e-458a-9589-7c9cc05b3ed5 None admin] RBAC: Adding query filter params (role.id=d729ee72a6d746959af49c5cab72ddd6, scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362, user.id=a43f8e46711349d4bcf7bab7f618f326) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:48.261626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b5140d-8e1e-458a-9589-7c9cc05b3ed5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:48.263129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b5140d-8e1e-458a-9589-7c9cc05b3ed5 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:48.263692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:48.275237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 47/102] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:26:47 2022] GET /identity/v3/role_assignments?user.id=a43f8e46711349d4bcf7bab7f618f326&scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362&role.id=d729ee72a6d746959af49c5cab72ddd6 => generated 608 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:48.276488 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae3878d-ee8f-40c0-ab85-8bb747385c6c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.276937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae3878d-ee8f-40c0-ab85-8bb747385c6c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.277357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae3878d-ee8f-40c0-ab85-8bb747385c6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.299204 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-fae3878d-ee8f-40c0-ab85-8bb747385c6c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:48.300705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-fae3878d-ee8f-40c0-ab85-8bb747385c6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:48.372140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d5859-f3ea-4a42-8f96-9184f657d6f8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:48.390561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 48/103] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:48.393093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae3878d-ee8f-40c0-ab85-8bb747385c6c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:48.415422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:48.416303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.416731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.417219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.418145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:48.418717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:48.420399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:48.436396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 56/104] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:26:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:48.439361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:48.442708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:48.530921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae3878d-ee8f-40c0-ab85-8bb747385c6c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:48.579099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:48.585194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df0e1e2-98db-48b9-b0de-2dbdebe59004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:48.586529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0e1e2-98db-48b9-b0de-2dbdebe59004 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.586993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0e1e2-98db-48b9-b0de-2dbdebe59004 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.587693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0e1e2-98db-48b9-b0de-2dbdebe59004 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.597308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abcd286-c557-480d-bb25-73a57e5e4799 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:48.603686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df0e1e2-98db-48b9-b0de-2dbdebe59004 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-1df0e1e2-98db-48b9-b0de-2dbdebe59004 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:48.610707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:48.611740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:48.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:48.614143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 49/105] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:48.617214 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb8e3e-050d-42cf-8453-e5604e5ec1ee None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.617710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb8e3e-050d-42cf-8453-e5604e5ec1ee None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.618186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb8e3e-050d-42cf-8453-e5604e5ec1ee None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.625205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 50/106] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:48 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:48.625205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bb8e3e-050d-42cf-8453-e5604e5ec1ee None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:48.642373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e224e5a-3985-487f-b74f-0545b94d43ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:48.643060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e224e5a-3985-487f-b74f-0545b94d43ff None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.643348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e224e5a-3985-487f-b74f-0545b94d43ff None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.643625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e224e5a-3985-487f-b74f-0545b94d43ff None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.659153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e224e5a-3985-487f-b74f-0545b94d43ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-0e224e5a-3985-487f-b74f-0545b94d43ff None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:48.665253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:48.666227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:48.667410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 57/107] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:26:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:48.673221 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e224e5a-3985-487f-b74f-0545b94d43ff None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:48.747961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:48.760404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bb8e3e-050d-42cf-8453-e5604e5ec1ee None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:48.792982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e224e5a-3985-487f-b74f-0545b94d43ff None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:48.806010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf5203-92c4-42ad-a0fd-a561f5cfb844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:48.806703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf5203-92c4-42ad-a0fd-a561f5cfb844 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.806993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf5203-92c4-42ad-a0fd-a561f5cfb844 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.807273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf5203-92c4-42ad-a0fd-a561f5cfb844 None admin] PATH_INFO: `/v3/users/a43f8e46711349d4bcf7bab7f618f326` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.808467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf5203-92c4-42ad-a0fd-a561f5cfb844 None admin] RBAC: Authorizing `identity:get_user(user_id=a43f8e46711349d4bcf7bab7f618f326)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:48.809975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf5203-92c4-42ad-a0fd-a561f5cfb844 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:48.811657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 51/108] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:48 2022] GET /identity/v3/users/a43f8e46711349d4bcf7bab7f618f326 => generated 275 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:48.815971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8462e5ce-1c21-4bbd-902b-22166c4c09e6 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.816440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8462e5ce-1c21-4bbd-902b-22166c4c09e6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.816895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8462e5ce-1c21-4bbd-902b-22166c4c09e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.833771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:48.834474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:48.834869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:48.835207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:48.836039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] RBAC: Adding query filter params (name=member) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:48.836408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:48.836751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8462e5ce-1c21-4bbd-902b-22166c4c09e6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:48.837932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:48.838024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8462e5ce-1c21-4bbd-902b-22166c4c09e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:48.848017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 58/109] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:26:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:48.849419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:48.863833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:48.924812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 52/110] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:48.924812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8462e5ce-1c21-4bbd-902b-22166c4c09e6 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:48.993412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3e0b99-4e49-43cb-b953-94895b63ac39 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.050276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce56eda-52ca-445a-a4cb-c724da3956bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.051409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce56eda-52ca-445a-a4cb-c724da3956bc None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.051889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce56eda-52ca-445a-a4cb-c724da3956bc None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.052373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce56eda-52ca-445a-a4cb-c724da3956bc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.054506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce56eda-52ca-445a-a4cb-c724da3956bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:49.057239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce56eda-52ca-445a-a4cb-c724da3956bc None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:49.062777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 59/111] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:49.062777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce56eda-52ca-445a-a4cb-c724da3956bc None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:49.063555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:49.077356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8462e5ce-1c21-4bbd-902b-22166c4c09e6 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.122530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.127425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.127425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.127425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.127425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:49.127425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:49.127425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:49.142761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 53/112] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:49.142761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:49.183585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:49.213657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce56eda-52ca-445a-a4cb-c724da3956bc None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.254124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f2446a-7fda-492e-a28d-6bf06767e9e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.254786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f2446a-7fda-492e-a28d-6bf06767e9e1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.255071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f2446a-7fda-492e-a28d-6bf06767e9e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.255346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f2446a-7fda-492e-a28d-6bf06767e9e1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.278934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2646c03f-e383-4ffa-8dbd-4209544a8f2a None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.316706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0143b40-2213-4f2c-a5d0-fff0479e067f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.317356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0143b40-2213-4f2c-a5d0-fff0479e067f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.317640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0143b40-2213-4f2c-a5d0-fff0479e067f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.317917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0143b40-2213-4f2c-a5d0-fff0479e067f None admin] PATH_INFO: `/v3/users/a43f8e46711349d4bcf7bab7f618f326` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.319107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0143b40-2213-4f2c-a5d0-fff0479e067f None admin] RBAC: Authorizing `identity:get_user(user_id=a43f8e46711349d4bcf7bab7f618f326)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:49.320576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0143b40-2213-4f2c-a5d0-fff0479e067f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:49.322391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 54/113] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:49 2022] GET /identity/v3/users/a43f8e46711349d4bcf7bab7f618f326 => generated 275 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:49.323108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0143b40-2213-4f2c-a5d0-fff0479e067f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:49.343728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f2446a-7fda-492e-a28d-6bf06767e9e1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:49.345283 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f2446a-7fda-492e-a28d-6bf06767e9e1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:49.347321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:49.432501 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:49.443442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0143b40-2213-4f2c-a5d0-fff0479e067f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.464425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 60/114] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:26:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:26:49.467814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f2446a-7fda-492e-a28d-6bf06767e9e1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:49.487063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec904cff-e43b-459d-afb6-93c5c1d620e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.487684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec904cff-e43b-459d-afb6-93c5c1d620e7 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.489722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec904cff-e43b-459d-afb6-93c5c1d620e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.489722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec904cff-e43b-459d-afb6-93c5c1d620e7 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.489722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec904cff-e43b-459d-afb6-93c5c1d620e7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:49.490458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec904cff-e43b-459d-afb6-93c5c1d620e7 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:49.519756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 55/115] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:26:49 2022] POST /identity/v3/projects => generated 302 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:49.520603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec904cff-e43b-459d-afb6-93c5c1d620e7 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:49.601183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f2446a-7fda-492e-a28d-6bf06767e9e1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.636144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:49.650461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b591be-ef21-4785-aa61-dfc358d8532a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.651852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b591be-ef21-4785-aa61-dfc358d8532a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.652572 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b591be-ef21-4785-aa61-dfc358d8532a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.653227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b591be-ef21-4785-aa61-dfc358d8532a None admin] PATH_INFO: `/v3/users/ded9b81a3e894fd88539e86bbeb17ee3` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.654924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b591be-ef21-4785-aa61-dfc358d8532a None admin] RBAC: Authorizing `identity:get_user(user_id=ded9b81a3e894fd88539e86bbeb17ee3)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:49.656977 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b591be-ef21-4785-aa61-dfc358d8532a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:49.660234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 61/116] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:49 2022] GET /identity/v3/users/ded9b81a3e894fd88539e86bbeb17ee3 => generated 247 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:49.661396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b591be-ef21-4785-aa61-dfc358d8532a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:49.691699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec904cff-e43b-459d-afb6-93c5c1d620e7 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.720604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:49.748411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5d4810-ae20-4035-9bb0-0e5487beaed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.749115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5d4810-ae20-4035-9bb0-0e5487beaed1 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.749398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5d4810-ae20-4035-9bb0-0e5487beaed1 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.749732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5d4810-ae20-4035-9bb0-0e5487beaed1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.762760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5d4810-ae20-4035-9bb0-0e5487beaed1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-2a5d4810-ae20-4035-9bb0-0e5487beaed1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:49.768652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:49.769553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:49.770305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 56/117] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:49.771082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5d4810-ae20-4035-9bb0-0e5487beaed1 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:49.794211 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b591be-ef21-4785-aa61-dfc358d8532a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.837411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951af795-6310-4496-95a4-f90bb5dd0747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.838043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951af795-6310-4496-95a4-f90bb5dd0747 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.838329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951af795-6310-4496-95a4-f90bb5dd0747 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.838663 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951af795-6310-4496-95a4-f90bb5dd0747 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.893985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5d4810-ae20-4035-9bb0-0e5487beaed1 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:49.915387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951af795-6310-4496-95a4-f90bb5dd0747 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:49.916987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951af795-6310-4496-95a4-f90bb5dd0747 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:49.947361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:49.955309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a53c319-ee11-49a4-af00-bb41e9a85d50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:49.956227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a53c319-ee11-49a4-af00-bb41e9a85d50 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:49.956651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a53c319-ee11-49a4-af00-bb41e9a85d50 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:49.957048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a53c319-ee11-49a4-af00-bb41e9a85d50 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:49.958633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a53c319-ee11-49a4-af00-bb41e9a85d50 None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:49.960777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a53c319-ee11-49a4-af00-bb41e9a85d50 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:49.967543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 57/118] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:49 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:49.967543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a53c319-ee11-49a4-af00-bb41e9a85d50 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:49.979040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:50.055691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 62/119] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:26:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:26:50.063762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951af795-6310-4496-95a4-f90bb5dd0747 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:50.109004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a53c319-ee11-49a4-af00-bb41e9a85d50 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:50.154044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05bfe13-b209-4eb8-afe1-184ee7b9af2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:50.154707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05bfe13-b209-4eb8-afe1-184ee7b9af2f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:50.154992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05bfe13-b209-4eb8-afe1-184ee7b9af2f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:50.155270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05bfe13-b209-4eb8-afe1-184ee7b9af2f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:50.203954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951af795-6310-4496-95a4-f90bb5dd0747 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:50.246638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:50.247210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:50.247849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:50.248137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:50.248597 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:50.249332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:50.249641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:50.249997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05bfe13-b209-4eb8-afe1-184ee7b9af2f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:50.251966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:50.253037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05bfe13-b209-4eb8-afe1-184ee7b9af2f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:50.263343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 63/120] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:26:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:50.264639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:50.280203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:50.390269 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 58/121] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:26:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:26:50.395397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05bfe13-b209-4eb8-afe1-184ee7b9af2f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:50.395485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c0b1cb-d22c-4851-8839-0306abc55a0e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:50.441464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:50.442098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:50.442374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:50.442678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:50.443419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] RBAC: Adding query filter params (role.id=66029780d52042b9a44109eb91c0d80e, scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362, user.id=a43f8e46711349d4bcf7bab7f618f326) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:50.443726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:50.445198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:50.454780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 64/122] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:26:50 2022] GET /identity/v3/role_assignments?user.id=a43f8e46711349d4bcf7bab7f618f326&scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362&role.id=66029780d52042b9a44109eb91c0d80e => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:50.465503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:50.517481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:50.521334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05bfe13-b209-4eb8-afe1-184ee7b9af2f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:50.573096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:50.598695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5985422-0ab8-44d0-ad62-a10fe1e41629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:50.600620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5985422-0ab8-44d0-ad62-a10fe1e41629 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:50.601425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5985422-0ab8-44d0-ad62-a10fe1e41629 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:50.602235 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5985422-0ab8-44d0-ad62-a10fe1e41629 None admin] PATH_INFO: `/v3/projects/d3942a754a534cdc8791e7c3faec8aaa` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:50.604906 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5985422-0ab8-44d0-ad62-a10fe1e41629 None admin] RBAC: Authorizing `identity:get_project(project_id=d3942a754a534cdc8791e7c3faec8aaa)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:50.607238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5985422-0ab8-44d0-ad62-a10fe1e41629 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:50.610234 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 59/123] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:50 2022] GET /identity/v3/projects/d3942a754a534cdc8791e7c3faec8aaa => generated 312 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:50.611792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5985422-0ab8-44d0-ad62-a10fe1e41629 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:50.620877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391ffbdd-65d0-4110-9850-57fd4da6eb12 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:50.679351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81a39a-9727-45ee-bdb1-0495cef0060f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:50.681931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81a39a-9727-45ee-bdb1-0495cef0060f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:50.681931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81a39a-9727-45ee-bdb1-0495cef0060f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:50.681931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee81a39a-9727-45ee-bdb1-0495cef0060f None admin] PATH_INFO: `/v3/users/3034490018664f398123e021928fbfa7` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:50.703155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee81a39a-9727-45ee-bdb1-0495cef0060f None admin] RBAC: Authorizing `identity:get_user(user_id=3034490018664f398123e021928fbfa7)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:50.705357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee81a39a-9727-45ee-bdb1-0495cef0060f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:50.707529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 65/124] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:50 2022] GET /identity/v3/users/3034490018664f398123e021928fbfa7 => generated 289 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:50.708131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81a39a-9727-45ee-bdb1-0495cef0060f None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:50.759823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5985422-0ab8-44d0-ad62-a10fe1e41629 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:50.801384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaf804f-d552-4b91-a62b-a78f57e7e983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:50.802260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaf804f-d552-4b91-a62b-a78f57e7e983 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:50.802717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaf804f-d552-4b91-a62b-a78f57e7e983 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:50.803159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaf804f-d552-4b91-a62b-a78f57e7e983 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:50.804295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaf804f-d552-4b91-a62b-a78f57e7e983 None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:50.805749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaf804f-d552-4b91-a62b-a78f57e7e983 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:50.807446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 60/125] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:50 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:50.808237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaf804f-d552-4b91-a62b-a78f57e7e983 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:50.859321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:50.878467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee81a39a-9727-45ee-bdb1-0495cef0060f None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:50.925777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaf804f-d552-4b91-a62b-a78f57e7e983 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:50.931250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:50.942783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:50.943867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:50.944325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:50.944785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:50.945993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] RBAC: Adding query filter params (role.id=d729ee72a6d746959af49c5cab72ddd6, scope.project.id=d3942a754a534cdc8791e7c3faec8aaa, user.id=a43f8e46711349d4bcf7bab7f618f326) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:50.946675 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:50.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:50.960929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cefa66-d46d-4da2-88db-551023a49b52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:50.966727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cefa66-d46d-4da2-88db-551023a49b52 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:50.966727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cefa66-d46d-4da2-88db-551023a49b52 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:50.966964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 66/126] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:26:50 2022] GET /identity/v3/role_assignments?user.id=a43f8e46711349d4bcf7bab7f618f326&scope.project.id=d3942a754a534cdc8791e7c3faec8aaa&role.id=d729ee72a6d746959af49c5cab72ddd6 => generated 608 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:50.967044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cefa66-d46d-4da2-88db-551023a49b52 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:50.967110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:50.970517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cefa66-d46d-4da2-88db-551023a49b52 None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:50.970517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cefa66-d46d-4da2-88db-551023a49b52 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:50.971625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 61/127] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:50 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:50.978064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cefa66-d46d-4da2-88db-551023a49b52 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:51.116698 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cefa66-d46d-4da2-88db-551023a49b52 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:51.118707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec780fa6-ef16-414e-a8d1-301a1dc6d137 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:51.120298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:51.163453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92c304c-e955-40bd-bdb4-6d07d4cba45a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:51.164110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c304c-e955-40bd-bdb4-6d07d4cba45a None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:51.164402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c304c-e955-40bd-bdb4-6d07d4cba45a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:51.164683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c304c-e955-40bd-bdb4-6d07d4cba45a None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/users/3034490018664f398123e021928fbfa7/roles/c65aae75f4b449d0adda4ee067622739` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:51.166715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92c304c-e955-40bd-bdb4-6d07d4cba45a None admin] RBAC: Authorizing `identity:create_grant(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362, user_id=3034490018664f398123e021928fbfa7, role_id=c65aae75f4b449d0adda4ee067622739)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:51.168192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92c304c-e955-40bd-bdb4-6d07d4cba45a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:51.168806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c337fc4-9b4f-42e4-8eea-590776cf00a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:51.169419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c337fc4-9b4f-42e4-8eea-590776cf00a1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:51.169702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c337fc4-9b4f-42e4-8eea-590776cf00a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:51.169974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c337fc4-9b4f-42e4-8eea-590776cf00a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:51.171497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c337fc4-9b4f-42e4-8eea-590776cf00a1 None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362, user.id=ded9b81a3e894fd88539e86bbeb17ee3) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:51.171497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c337fc4-9b4f-42e4-8eea-590776cf00a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:51.174091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c337fc4-9b4f-42e4-8eea-590776cf00a1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:51.181532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 67/128] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:26:50 2022] GET /identity/v3/role_assignments?user.id=ded9b81a3e894fd88539e86bbeb17ee3&scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 608 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:51.184876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 62/129] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:26:50 2022] PUT /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/users/3034490018664f398123e021928fbfa7/roles/c65aae75f4b449d0adda4ee067622739 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:26:51.814829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce27906-c0ad-4d27-a567-8e0458618ece None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:51.815043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce27906-c0ad-4d27-a567-8e0458618ece None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:51.815321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce27906-c0ad-4d27-a567-8e0458618ece None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:51.816692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 68/130] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:51.822089 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070b65a4-08d3-44c8-bda1-98b03b0d7606 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:51.822384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070b65a4-08d3-44c8-bda1-98b03b0d7606 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:51.822704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070b65a4-08d3-44c8-bda1-98b03b0d7606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:51.841267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-070b65a4-08d3-44c8-bda1-98b03b0d7606 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:51.842475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-070b65a4-08d3-44c8-bda1-98b03b0d7606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:51.909866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 63/131] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:52.202979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce27906-c0ad-4d27-a567-8e0458618ece None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:52.264471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:52.308897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce27906-c0ad-4d27-a567-8e0458618ece None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:52.343391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94629659-699a-49df-8a86-f198dae265e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:52.344166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94629659-699a-49df-8a86-f198dae265e3 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:52.344464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94629659-699a-49df-8a86-f198dae265e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:52.344744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94629659-699a-49df-8a86-f198dae265e3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:52.345865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94629659-699a-49df-8a86-f198dae265e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:52.347379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94629659-699a-49df-8a86-f198dae265e3 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:52.349442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 69/132] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:52.354796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070b65a4-08d3-44c8-bda1-98b03b0d7606 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:52.457339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:52.465458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070b65a4-08d3-44c8-bda1-98b03b0d7606 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:52.506872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839bbce1-3cee-4984-abd8-63a897df4810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:52.507518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839bbce1-3cee-4984-abd8-63a897df4810 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:52.507806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839bbce1-3cee-4984-abd8-63a897df4810 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:52.508083 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839bbce1-3cee-4984-abd8-63a897df4810 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:52.508782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839bbce1-3cee-4984-abd8-63a897df4810 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:52.510290 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839bbce1-3cee-4984-abd8-63a897df4810 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:52.534167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 64/133] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:26:52 2022] POST /identity/v3/users => generated 283 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:53.469043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6b6e7-647f-4315-a3bf-08fd98fee31a None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:53.469672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6b6e7-647f-4315-a3bf-08fd98fee31a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:53.469983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6b6e7-647f-4315-a3bf-08fd98fee31a None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:53.471642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 70/134] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:53.477500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512d47d-4c90-45ae-8fbc-95834f094da7 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:53.478342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512d47d-4c90-45ae-8fbc-95834f094da7 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:53.478846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512d47d-4c90-45ae-8fbc-95834f094da7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:53.498381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-4512d47d-4c90-45ae-8fbc-95834f094da7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:53.499959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4512d47d-4c90-45ae-8fbc-95834f094da7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:53.579563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 65/135] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:53.850371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6b6e7-647f-4315-a3bf-08fd98fee31a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:53.955927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6b6e7-647f-4315-a3bf-08fd98fee31a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:53.987280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:53.998823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780853b-ce93-4091-befc-6ea812333599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:53.999551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780853b-ce93-4091-befc-6ea812333599 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:53.999871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780853b-ce93-4091-befc-6ea812333599 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:54.000704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780853b-ce93-4091-befc-6ea812333599 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:54.012956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1780853b-ce93-4091-befc-6ea812333599 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-1780853b-ce93-4091-befc-6ea812333599 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:54.018847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:54.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:54.021201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 71/136] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:53 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:54.026359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4512d47d-4c90-45ae-8fbc-95834f094da7 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:54.118272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:54.139122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4512d47d-4c90-45ae-8fbc-95834f094da7 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:54.174940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:54.175568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:54.175875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:54.176131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:54.176810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] RBAC: Adding query filter params (name=reader) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:54.177117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:54.178519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:54.187128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 66/137] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:26:54 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:54.193725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780853b-ce93-4091-befc-6ea812333599 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:54.316758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780853b-ce93-4091-befc-6ea812333599 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:54.357915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a9fb4a-130b-45d4-aa4f-9494e0c9f3f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:54.358600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a9fb4a-130b-45d4-aa4f-9494e0c9f3f0 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:54.358928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a9fb4a-130b-45d4-aa4f-9494e0c9f3f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:54.359277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a9fb4a-130b-45d4-aa4f-9494e0c9f3f0 None admin] PATH_INFO: `/v3/users/3034490018664f398123e021928fbfa7` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:54.361098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a9fb4a-130b-45d4-aa4f-9494e0c9f3f0 None admin] RBAC: Authorizing `identity:get_user(user_id=3034490018664f398123e021928fbfa7)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:54.362571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a9fb4a-130b-45d4-aa4f-9494e0c9f3f0 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:54.364388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 72/138] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:54 2022] GET /identity/v3/users/3034490018664f398123e021928fbfa7 => generated 289 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:54.369809 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:54.475497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2169c6-c999-4dcf-9469-0d22e8f9b4ca None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:54.512011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a341de-daae-4449-b314-5848da9e5371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:54.512661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a341de-daae-4449-b314-5848da9e5371 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:54.512914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a341de-daae-4449-b314-5848da9e5371 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:54.513199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a341de-daae-4449-b314-5848da9e5371 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:54.528113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:54.592408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a341de-daae-4449-b314-5848da9e5371 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:54.593983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a341de-daae-4449-b314-5848da9e5371 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:54.715467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 67/139] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:26:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:26:54.721959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a9fb4a-130b-45d4-aa4f-9494e0c9f3f0 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:54.783379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:54.835471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a9fb4a-130b-45d4-aa4f-9494e0c9f3f0 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:54.873388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a54c4b-998e-48f0-9914-91b4f34c5cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:54.874465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a54c4b-998e-48f0-9914-91b4f34c5cca None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:54.874981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a54c4b-998e-48f0-9914-91b4f34c5cca None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:54.876137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a54c4b-998e-48f0-9914-91b4f34c5cca None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:54.877735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a54c4b-998e-48f0-9914-91b4f34c5cca None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:54.879502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a54c4b-998e-48f0-9914-91b4f34c5cca None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:54.881845 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 73/140] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:54 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:54.891795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a341de-daae-4449-b314-5848da9e5371 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:54.921360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b1bda-d440-42fb-9dfa-4663c5dfb3a9 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:54.922674 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b1bda-d440-42fb-9dfa-4663c5dfb3a9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:54.923280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b1bda-d440-42fb-9dfa-4663c5dfb3a9 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:54.924922 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 74/141] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:26:54.938520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897e56f-d049-44ad-923f-9afb25e94800 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:54.938520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897e56f-d049-44ad-923f-9afb25e94800 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:54.938520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897e56f-d049-44ad-923f-9afb25e94800 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:54.940917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:54.956794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b897e56f-d049-44ad-923f-9afb25e94800 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:54.958361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b897e56f-d049-44ad-923f-9afb25e94800 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:55.012414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a341de-daae-4449-b314-5848da9e5371 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:55.045012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 75/142] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:55.062776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41904b6-becb-4b3f-b4d5-f0432bf379b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:55.063404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98a6462-9269-4122-a2c1-886e77f99f79 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.063861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98a6462-9269-4122-a2c1-886e77f99f79 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.064117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41904b6-becb-4b3f-b4d5-f0432bf379b6 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.064333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98a6462-9269-4122-a2c1-886e77f99f79 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.064718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41904b6-becb-4b3f-b4d5-f0432bf379b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.065154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41904b6-becb-4b3f-b4d5-f0432bf379b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.066133 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41904b6-becb-4b3f-b4d5-f0432bf379b6 None admin] RBAC: Adding query filter params (role.id=c65aae75f4b449d0adda4ee067622739, scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362, user.id=3034490018664f398123e021928fbfa7) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:55.066465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 76/143] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:26:55.066886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41904b6-becb-4b3f-b4d5-f0432bf379b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:55.068640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41904b6-becb-4b3f-b4d5-f0432bf379b6 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:55.072861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e974cd9f-2401-4c46-986e-43ed682e26c2 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.072861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e974cd9f-2401-4c46-986e-43ed682e26c2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.073265 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e974cd9f-2401-4c46-986e-43ed682e26c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.077383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 68/144] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:26:54 2022] GET /identity/v3/role_assignments?user.id=3034490018664f398123e021928fbfa7&scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362&role.id=c65aae75f4b449d0adda4ee067622739 => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:55.096972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e974cd9f-2401-4c46-986e-43ed682e26c2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:55.098973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e974cd9f-2401-4c46-986e-43ed682e26c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:55.158072 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:55.183063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2a07e-ec21-45ef-9791-550f6f68040a None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.183412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2a07e-ec21-45ef-9791-550f6f68040a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.183619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2a07e-ec21-45ef-9791-550f6f68040a None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.184976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 69/145] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:26:55.185292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 77/146] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:55.191339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9cbd7f-2249-42ca-9078-cd3ca9c3ba7e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.191637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9cbd7f-2249-42ca-9078-cd3ca9c3ba7e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.191900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9cbd7f-2249-42ca-9078-cd3ca9c3ba7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.213229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-cb9cbd7f-2249-42ca-9078-cd3ca9c3ba7e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:55.214464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-cb9cbd7f-2249-42ca-9078-cd3ca9c3ba7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:55.289876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 70/147] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:55.349651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e974cd9f-2401-4c46-986e-43ed682e26c2 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:55.457667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e974cd9f-2401-4c46-986e-43ed682e26c2 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:55.458726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9cbd7f-2249-42ca-9078-cd3ca9c3ba7e None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:55.494945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:55.496788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b23697-5807-4e4d-b3ac-f67908d3223c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:55.497502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b23697-5807-4e4d-b3ac-f67908d3223c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.497807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b23697-5807-4e4d-b3ac-f67908d3223c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.498086 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b23697-5807-4e4d-b3ac-f67908d3223c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.509687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b23697-5807-4e4d-b3ac-f67908d3223c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-25b23697-5807-4e4d-b3ac-f67908d3223c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:55.516844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:55.518468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 78/148] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:26:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:55.523147 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b23697-5807-4e4d-b3ac-f67908d3223c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:55.570995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9cbd7f-2249-42ca-9078-cd3ca9c3ba7e None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:55.577713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:55.607112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9f4cb3-1747-45f3-b3a9-35f65ca16eaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:55.607751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f4cb3-1747-45f3-b3a9-35f65ca16eaa None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.608023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f4cb3-1747-45f3-b3a9-35f65ca16eaa None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.608292 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9f4cb3-1747-45f3-b3a9-35f65ca16eaa None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.619434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9f4cb3-1747-45f3-b3a9-35f65ca16eaa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-4d9f4cb3-1747-45f3-b3a9-35f65ca16eaa None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:55.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:55.626041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:55.626983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 71/149] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:26:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:55.627672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9f4cb3-1747-45f3-b3a9-35f65ca16eaa None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:55.640453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b23697-5807-4e4d-b3ac-f67908d3223c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:55.675783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:55.676394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.676672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.676943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.677627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:55.677908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:55.679339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:55.687879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 79/150] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:55.688789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:55.727166 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:55.733626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9f4cb3-1747-45f3-b3a9-35f65ca16eaa None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:55.767981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840ef6dc-80e4-4e66-99f5-ea38f1bf9011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:55.768613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ef6dc-80e4-4e66-99f5-ea38f1bf9011 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.768883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ef6dc-80e4-4e66-99f5-ea38f1bf9011 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.769158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ef6dc-80e4-4e66-99f5-ea38f1bf9011 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.770249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ef6dc-80e4-4e66-99f5-ea38f1bf9011 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:55.771735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ef6dc-80e4-4e66-99f5-ea38f1bf9011 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:55.773294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 72/151] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:55.773294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840ef6dc-80e4-4e66-99f5-ea38f1bf9011 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:55.791942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156cc1ab-377c-4578-95b0-46ec054c70e5 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:55.794311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:55.825754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:55.826287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.826602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.826884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.827552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:55.827861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:55.829938 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:55.838002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 80/152] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:26:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:55.839042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:55.877614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840ef6dc-80e4-4e66-99f5-ea38f1bf9011 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:55.913123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4255a2e5-32fd-4ce1-b340-d3a04ccb7986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:55.913774 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4255a2e5-32fd-4ce1-b340-d3a04ccb7986 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.914052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4255a2e5-32fd-4ce1-b340-d3a04ccb7986 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.914323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4255a2e5-32fd-4ce1-b340-d3a04ccb7986 None admin] PATH_INFO: `/v3/users/5695028eca23410ebfb7f516d585ac55` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.929902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4255a2e5-32fd-4ce1-b340-d3a04ccb7986 None admin] RBAC: Authorizing `identity:get_user(user_id=5695028eca23410ebfb7f516d585ac55)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:55.931484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4255a2e5-32fd-4ce1-b340-d3a04ccb7986 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:55.933384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 73/153] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:55 2022] GET /identity/v3/users/5695028eca23410ebfb7f516d585ac55 => generated 283 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:55.933384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4255a2e5-32fd-4ce1-b340-d3a04ccb7986 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:55.948701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5a64c-e786-4588-a429-1f758472cd4a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:55.962073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:55.985741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4ffc49-9fd6-458b-9238-cb3c487aee2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:55.986348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4ffc49-9fd6-458b-9238-cb3c487aee2b None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:55.986735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4ffc49-9fd6-458b-9238-cb3c487aee2b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:55.987008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4ffc49-9fd6-458b-9238-cb3c487aee2b None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:55.987657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4ffc49-9fd6-458b-9238-cb3c487aee2b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:55.989112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4ffc49-9fd6-458b-9238-cb3c487aee2b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:56.016813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 81/154] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:26:55 2022] POST /identity/v3/users => generated 297 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:56.019467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4ffc49-9fd6-458b-9238-cb3c487aee2b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:56.045117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4255a2e5-32fd-4ce1-b340-d3a04ccb7986 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:56.051818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:56.086881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fd896a-2fe7-4788-a3cc-45fc24a8dac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:56.087448 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fd896a-2fe7-4788-a3cc-45fc24a8dac7 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:56.087730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fd896a-2fe7-4788-a3cc-45fc24a8dac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:56.088008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fd896a-2fe7-4788-a3cc-45fc24a8dac7 None admin] PATH_INFO: `/v3/users/5695028eca23410ebfb7f516d585ac55` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:56.089196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fd896a-2fe7-4788-a3cc-45fc24a8dac7 None admin] RBAC: Authorizing `identity:get_user(user_id=5695028eca23410ebfb7f516d585ac55)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:56.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fd896a-2fe7-4788-a3cc-45fc24a8dac7 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:56.092598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 74/155] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:55 2022] GET /identity/v3/users/5695028eca23410ebfb7f516d585ac55 => generated 283 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:56.099085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fd896a-2fe7-4788-a3cc-45fc24a8dac7 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:56.152575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4ffc49-9fd6-458b-9238-cb3c487aee2b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:56.195225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9819e84-e40c-46fd-b885-0fe676198054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:56.195875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9819e84-e40c-46fd-b885-0fe676198054 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:56.196152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9819e84-e40c-46fd-b885-0fe676198054 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:56.196426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9819e84-e40c-46fd-b885-0fe676198054 None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:56.197559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9819e84-e40c-46fd-b885-0fe676198054 None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:56.199141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9819e84-e40c-46fd-b885-0fe676198054 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:56.201059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 82/156] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:56 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:56.206765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9819e84-e40c-46fd-b885-0fe676198054 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:56.210656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:56.224851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fd896a-2fe7-4788-a3cc-45fc24a8dac7 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:56.267445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a7e94f-7579-42d4-b926-45072473e1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:56.267954 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a7e94f-7579-42d4-b926-45072473e1bf None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:56.268239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a7e94f-7579-42d4-b926-45072473e1bf None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:56.268515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a7e94f-7579-42d4-b926-45072473e1bf None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:56.269771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a7e94f-7579-42d4-b926-45072473e1bf None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:56.272357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a7e94f-7579-42d4-b926-45072473e1bf None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:56.273724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 75/157] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:26:56 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:26:56.280676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a7e94f-7579-42d4-b926-45072473e1bf None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:56.319469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:56.335272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9819e84-e40c-46fd-b885-0fe676198054 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:56.386719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5c9bec-94ef-4e37-8011-ece56893735a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:56.387311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5c9bec-94ef-4e37-8011-ece56893735a None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:56.387590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5c9bec-94ef-4e37-8011-ece56893735a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:56.387864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5c9bec-94ef-4e37-8011-ece56893735a None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/users/5695028eca23410ebfb7f516d585ac55/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:56.389777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5c9bec-94ef-4e37-8011-ece56893735a None admin] RBAC: Authorizing `identity:create_grant(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, user_id=5695028eca23410ebfb7f516d585ac55, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:56.391839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5c9bec-94ef-4e37-8011-ece56893735a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:56.407532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 83/158] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:26:56 2022] PUT /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/users/5695028eca23410ebfb7f516d585ac55/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:26:56.418445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a7e94f-7579-42d4-b926-45072473e1bf None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:56.479671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262d0f1c-c309-4a26-979a-06081c19db0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:56.480839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d0f1c-c309-4a26-979a-06081c19db0f None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:56.481386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d0f1c-c309-4a26-979a-06081c19db0f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:56.481951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d0f1c-c309-4a26-979a-06081c19db0f None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/users/5695028eca23410ebfb7f516d585ac55/roles/66029780d52042b9a44109eb91c0d80e` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:56.485650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262d0f1c-c309-4a26-979a-06081c19db0f None admin] RBAC: Authorizing `identity:create_grant(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, user_id=5695028eca23410ebfb7f516d585ac55, role_id=66029780d52042b9a44109eb91c0d80e)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:56.487902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262d0f1c-c309-4a26-979a-06081c19db0f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:56.505632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 76/159] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:26:56 2022] PUT /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/users/5695028eca23410ebfb7f516d585ac55/roles/66029780d52042b9a44109eb91c0d80e => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:26:58.622588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5ca2f1-fde2-432c-b79b-6ae342cf6de0 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:58.623252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5ca2f1-fde2-432c-b79b-6ae342cf6de0 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:58.623252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5ca2f1-fde2-432c-b79b-6ae342cf6de0 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:58.624814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 84/160] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:58.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c7230-0491-4613-bc44-01e05a576759 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:58.632159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c7230-0491-4613-bc44-01e05a576759 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:58.632656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c7230-0491-4613-bc44-01e05a576759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:58.655296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ad7c7230-0491-4613-bc44-01e05a576759 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:58.657102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ad7c7230-0491-4613-bc44-01e05a576759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:58.668761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5855c024-58e4-4064-8214-744ace9ebb47 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:58.669368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5855c024-58e4-4064-8214-744ace9ebb47 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:58.669815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5855c024-58e4-4064-8214-744ace9ebb47 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:58.672715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 85/161] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:58.675061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:58.683640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2991a4-9dc7-45b5-9fd4-3facfddb9cb5 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:58.685053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2991a4-9dc7-45b5-9fd4-3facfddb9cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:58.686025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2991a4-9dc7-45b5-9fd4-3facfddb9cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:58.713906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-3a2991a4-9dc7-45b5-9fd4-3facfddb9cb5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:58.715364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3a2991a4-9dc7-45b5-9fd4-3facfddb9cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:58.746759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 77/162] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:58.755957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34d975-1759-4e86-ba77-98f1f074bb41 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:58.756244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34d975-1759-4e86-ba77-98f1f074bb41 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:58.756528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34d975-1759-4e86-ba77-98f1f074bb41 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:58.762667 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 78/163] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:58.763901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9943f6-66b2-4e91-90bd-00e3b8f23401 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:58.764154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9943f6-66b2-4e91-90bd-00e3b8f23401 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:58.764433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9943f6-66b2-4e91-90bd-00e3b8f23401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:58.788087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-fa9943f6-66b2-4e91-90bd-00e3b8f23401 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:58.789283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-fa9943f6-66b2-4e91-90bd-00e3b8f23401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:58.830176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 86/164] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:58.864280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 79/165] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:26:59.013989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acc821f-5a3d-4693-a899-886d18bb4ece None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.014672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acc821f-5a3d-4693-a899-886d18bb4ece None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.015152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acc821f-5a3d-4693-a899-886d18bb4ece None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.016934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 87/166] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:26:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:26:59.022853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e76ed-f657-414d-be68-a455827922a0 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.023429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e76ed-f657-414d-be68-a455827922a0 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.023896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e76ed-f657-414d-be68-a455827922a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.044324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2b3e76ed-f657-414d-be68-a455827922a0 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:26:59.045848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2b3e76ed-f657-414d-be68-a455827922a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:26:59.057452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acc821f-5a3d-4693-a899-886d18bb4ece None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.118828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 80/167] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:26:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:26:59.119607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3e76ed-f657-414d-be68-a455827922a0 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.132663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:59.212260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acc821f-5a3d-4693-a899-886d18bb4ece None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:59.219785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:59.234890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3e76ed-f657-414d-be68-a455827922a0 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:59.264543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6c056d-56fb-4325-8d8a-0772e8c4572a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:59.265246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6c056d-56fb-4325-8d8a-0772e8c4572a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.265568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6c056d-56fb-4325-8d8a-0772e8c4572a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.266386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6c056d-56fb-4325-8d8a-0772e8c4572a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.274046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe2f87e-2736-473e-af04-71374ef2a865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:59.274704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe2f87e-2736-473e-af04-71374ef2a865 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.274980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe2f87e-2736-473e-af04-71374ef2a865 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.275253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe2f87e-2736-473e-af04-71374ef2a865 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.276358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe2f87e-2736-473e-af04-71374ef2a865 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:59.277806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe2f87e-2736-473e-af04-71374ef2a865 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:59.278633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6c056d-56fb-4325-8d8a-0772e8c4572a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:59.279580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 81/168] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:59.280235 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe2f87e-2736-473e-af04-71374ef2a865 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-1e6c056d-56fb-4325-8d8a-0772e8c4572a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:59.284893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:59.285960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:26:59.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:59.289142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 88/169] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:59.289142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6c056d-56fb-4325-8d8a-0772e8c4572a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.377013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:59.402749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe2f87e-2736-473e-af04-71374ef2a865 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:59.422967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6c056d-56fb-4325-8d8a-0772e8c4572a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:59.440297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507d4fb4-744c-4234-a145-64390fb00884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:59.440960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507d4fb4-744c-4234-a145-64390fb00884 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.441246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507d4fb4-744c-4234-a145-64390fb00884 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.441539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507d4fb4-744c-4234-a145-64390fb00884 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.453568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507d4fb4-744c-4234-a145-64390fb00884 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-507d4fb4-744c-4234-a145-64390fb00884 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:59.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:59.459391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:26:59.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 82/170] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:59.460857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507d4fb4-744c-4234-a145-64390fb00884 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.464627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c70c6c-a1ac-4209-b953-30fed2968d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:59.465261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c70c6c-a1ac-4209-b953-30fed2968d81 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.465539 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c70c6c-a1ac-4209-b953-30fed2968d81 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.466677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c70c6c-a1ac-4209-b953-30fed2968d81 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.466677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c70c6c-a1ac-4209-b953-30fed2968d81 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:59.467973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c70c6c-a1ac-4209-b953-30fed2968d81 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:59.480277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:59.492697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 89/171] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:26:59 2022] POST /identity/v3/users => generated 297 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:26:59.494686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c70c6c-a1ac-4209-b953-30fed2968d81 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.601915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507d4fb4-744c-4234-a145-64390fb00884 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:59.631642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c70c6c-a1ac-4209-b953-30fed2968d81 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:59.643954 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:59.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.645027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.645302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.645985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:59.646289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:59.647838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:59.650507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:59.659255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 83/172] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:59.660319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.681451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4abd8c-4f39-4ee5-8d9d-d6122a778e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:59.682108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4abd8c-4f39-4ee5-8d9d-d6122a778e49 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.682584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4abd8c-4f39-4ee5-8d9d-d6122a778e49 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.682884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4abd8c-4f39-4ee5-8d9d-d6122a778e49 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.697577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4abd8c-4f39-4ee5-8d9d-d6122a778e49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-ea4abd8c-4f39-4ee5-8d9d-d6122a778e49 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:26:59.705190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:26:59.706163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:26:59.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 90/173] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:26:59.707638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4abd8c-4f39-4ee5-8d9d-d6122a778e49 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.790296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:59.790657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1415c00-0365-4630-8359-35b01dad8e04 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:59.831402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:59.832030 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.832317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.832591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.833292 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] RBAC: Adding query filter params (name=member) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:26:59.833593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:59.835524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:59.845716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 84/174] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:59.847099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.852339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4abd8c-4f39-4ee5-8d9d-d6122a778e49 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:26:59.900851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ca6936-5a2b-45f4-9e1c-a75f311d42ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:26:59.901409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ca6936-5a2b-45f4-9e1c-a75f311d42ee None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:26:59.901687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ca6936-5a2b-45f4-9e1c-a75f311d42ee None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:26:59.901957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ca6936-5a2b-45f4-9e1c-a75f311d42ee None admin] PATH_INFO: `/v3/users/5695028eca23410ebfb7f516d585ac55` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:26:59.903144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ca6936-5a2b-45f4-9e1c-a75f311d42ee None admin] RBAC: Authorizing `identity:get_user(user_id=5695028eca23410ebfb7f516d585ac55)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:26:59.905101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ca6936-5a2b-45f4-9e1c-a75f311d42ee None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:26:59.907995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 91/175] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/users/5695028eca23410ebfb7f516d585ac55 => generated 283 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:26:59.909323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ca6936-5a2b-45f4-9e1c-a75f311d42ee None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:26:59.930177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:26:59.994040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa1c3fa-3fdd-4173-b056-b24a440ef75f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:00.034805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:00.035422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:00.035699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:00.035969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:00.036642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:00.037037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:00.038338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:00.046596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ca6936-5a2b-45f4-9e1c-a75f311d42ee None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:00.048187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 85/176] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:00.048867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:00.087513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:00.108761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b55269-01be-460c-8015-f719fa2a8295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:00.110399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b55269-01be-460c-8015-f719fa2a8295 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:00.110728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b55269-01be-460c-8015-f719fa2a8295 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:00.111009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b55269-01be-460c-8015-f719fa2a8295 None admin] PATH_INFO: `/v3/users/8e34c04d1dc140318b4b218ac6d6dbcd` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:00.129183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b55269-01be-460c-8015-f719fa2a8295 None admin] RBAC: Authorizing `identity:get_user(user_id=8e34c04d1dc140318b4b218ac6d6dbcd)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:00.130690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b55269-01be-460c-8015-f719fa2a8295 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:00.132714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 92/177] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:26:59 2022] GET /identity/v3/users/8e34c04d1dc140318b4b218ac6d6dbcd => generated 297 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:00.133631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b55269-01be-460c-8015-f719fa2a8295 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:00.195573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd7e38d-05d6-463c-b766-8d935b3202b2 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:00.226915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:00.245683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87955ac1-d9c9-4568-88db-4f3ca2457554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:00.246311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87955ac1-d9c9-4568-88db-4f3ca2457554 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:00.246666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87955ac1-d9c9-4568-88db-4f3ca2457554 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:00.246942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87955ac1-d9c9-4568-88db-4f3ca2457554 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:00.268549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b55269-01be-460c-8015-f719fa2a8295 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:00.313535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f893136c-ec4b-4eca-b573-72c545b7c147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:00.314935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893136c-ec4b-4eca-b573-72c545b7c147 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:00.315453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893136c-ec4b-4eca-b573-72c545b7c147 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:00.315959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893136c-ec4b-4eca-b573-72c545b7c147 None admin] PATH_INFO: `/v3/users/5695028eca23410ebfb7f516d585ac55` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:00.318013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f893136c-ec4b-4eca-b573-72c545b7c147 None admin] RBAC: Authorizing `identity:get_user(user_id=5695028eca23410ebfb7f516d585ac55)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:00.320307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f893136c-ec4b-4eca-b573-72c545b7c147 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:00.322652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 93/178] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:00 2022] GET /identity/v3/users/5695028eca23410ebfb7f516d585ac55 => generated 283 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:00.324514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f893136c-ec4b-4eca-b573-72c545b7c147 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:00.336119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87955ac1-d9c9-4568-88db-4f3ca2457554 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:00.337593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87955ac1-d9c9-4568-88db-4f3ca2457554 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:00.391253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:00.460659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f893136c-ec4b-4eca-b573-72c545b7c147 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:00.476779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 86/179] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:00.479437 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87955ac1-d9c9-4568-88db-4f3ca2457554 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:00.511483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:00.517101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467a989a-329f-4bfb-973d-1f4fdc5d894d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:00.517776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a989a-329f-4bfb-973d-1f4fdc5d894d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:00.518056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a989a-329f-4bfb-973d-1f4fdc5d894d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:00.518329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467a989a-329f-4bfb-973d-1f4fdc5d894d None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:00.519686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467a989a-329f-4bfb-973d-1f4fdc5d894d None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:00.521285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467a989a-329f-4bfb-973d-1f4fdc5d894d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:00.523057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 94/180] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:00 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:00.524266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467a989a-329f-4bfb-973d-1f4fdc5d894d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:00.613396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87955ac1-d9c9-4568-88db-4f3ca2457554 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:00.655349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d14564-0ba7-4a98-bfdf-486cce2f1e48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:00.656001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467a989a-329f-4bfb-973d-1f4fdc5d894d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:00.657920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d14564-0ba7-4a98-bfdf-486cce2f1e48 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:00.658453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d14564-0ba7-4a98-bfdf-486cce2f1e48 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:00.659004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d14564-0ba7-4a98-bfdf-486cce2f1e48 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:00.679406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:00.704017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95abaeaa-e993-464b-a88b-dc20dd6857ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:00.704727 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95abaeaa-e993-464b-a88b-dc20dd6857ee None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:00.705024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95abaeaa-e993-464b-a88b-dc20dd6857ee None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:00.705303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95abaeaa-e993-464b-a88b-dc20dd6857ee None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:00.707182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95abaeaa-e993-464b-a88b-dc20dd6857ee None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:00.709419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95abaeaa-e993-464b-a88b-dc20dd6857ee None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:00.711866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 95/181] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:00 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:00.715367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95abaeaa-e993-464b-a88b-dc20dd6857ee None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:00.758719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d14564-0ba7-4a98-bfdf-486cce2f1e48 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:00.761748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d14564-0ba7-4a98-bfdf-486cce2f1e48 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:00.799519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:00.846323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95abaeaa-e993-464b-a88b-dc20dd6857ee None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:00.897357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675cc239-4328-4820-bbba-9b96d6acfe30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:00.898185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675cc239-4328-4820-bbba-9b96d6acfe30 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:00.898535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675cc239-4328-4820-bbba-9b96d6acfe30 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:00.898887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675cc239-4328-4820-bbba-9b96d6acfe30 None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/users/8e34c04d1dc140318b4b218ac6d6dbcd/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:00.899582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 87/182] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:00.900672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d14564-0ba7-4a98-bfdf-486cce2f1e48 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:00.901074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675cc239-4328-4820-bbba-9b96d6acfe30 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, user_id=8e34c04d1dc140318b4b218ac6d6dbcd, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:00.902740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675cc239-4328-4820-bbba-9b96d6acfe30 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:00.921082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 96/183] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:00 2022] PUT /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/users/8e34c04d1dc140318b4b218ac6d6dbcd/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:00.923093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675cc239-4328-4820-bbba-9b96d6acfe30 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:00.991027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:01.026734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d14564-0ba7-4a98-bfdf-486cce2f1e48 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:01.056745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:01.061383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675cc239-4328-4820-bbba-9b96d6acfe30 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:01.092332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:01.092963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:01.093236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:01.093503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:01.094343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, user.id=5695028eca23410ebfb7f516d585ac55) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:01.094674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:01.096146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:01.107908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 88/184] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:00 2022] GET /identity/v3/role_assignments?user.id=5695028eca23410ebfb7f516d585ac55&scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 608 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:01.109746 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320a2b9b-62f9-4942-8458-d77347730b36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:01.110422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320a2b9b-62f9-4942-8458-d77347730b36 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:01.110859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320a2b9b-62f9-4942-8458-d77347730b36 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:01.111142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320a2b9b-62f9-4942-8458-d77347730b36 None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:01.112465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320a2b9b-62f9-4942-8458-d77347730b36 None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:01.113972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320a2b9b-62f9-4942-8458-d77347730b36 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:01.116216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 97/185] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:00 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:01.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:01.268251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87572839-1658-44f5-bd99-c3b3d00543c6 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:01.312164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89fe367-86c5-452f-9dcb-2f2cad03ded2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:01.313378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89fe367-86c5-452f-9dcb-2f2cad03ded2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:01.313669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89fe367-86c5-452f-9dcb-2f2cad03ded2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:01.313997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89fe367-86c5-452f-9dcb-2f2cad03ded2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:01.314761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89fe367-86c5-452f-9dcb-2f2cad03ded2 None admin] RBAC: Adding query filter params (role.id=66029780d52042b9a44109eb91c0d80e, scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, user.id=5695028eca23410ebfb7f516d585ac55) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:01.315064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89fe367-86c5-452f-9dcb-2f2cad03ded2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:01.316515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89fe367-86c5-452f-9dcb-2f2cad03ded2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:01.322862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 89/186] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:01 2022] GET /identity/v3/role_assignments?user.id=5695028eca23410ebfb7f516d585ac55&scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca&role.id=66029780d52042b9a44109eb91c0d80e => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:01.939327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0ece1-ea3f-4b4c-819b-69bbebbee078 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:01.939831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0ece1-ea3f-4b4c-819b-69bbebbee078 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:01.939831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0ece1-ea3f-4b4c-819b-69bbebbee078 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:01.941309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 98/187] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:01.946414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd752d7-d6e4-4039-a647-27f995b26db4 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:01.946852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd752d7-d6e4-4039-a647-27f995b26db4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:01.947045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd752d7-d6e4-4039-a647-27f995b26db4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:01.965777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-acd752d7-d6e4-4039-a647-27f995b26db4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:01.967012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-acd752d7-d6e4-4039-a647-27f995b26db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:02.019678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:02.059431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 90/188] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:02.276060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d86c1e-4b69-4d76-b8e0-cc39266af6f8 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:02.276427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d86c1e-4b69-4d76-b8e0-cc39266af6f8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:02.276695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d86c1e-4b69-4d76-b8e0-cc39266af6f8 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:02.278407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 99/189] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:02.284010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cdd8a-ad2b-49e1-ab5a-661b55f01c5c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:02.284363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cdd8a-ad2b-49e1-ab5a-661b55f01c5c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:02.284754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cdd8a-ad2b-49e1-ab5a-661b55f01c5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:02.303629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e33cdd8a-ad2b-49e1-ab5a-661b55f01c5c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:02.304810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e33cdd8a-ad2b-49e1-ab5a-661b55f01c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:02.338167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d86c1e-4b69-4d76-b8e0-cc39266af6f8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:02.376366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 91/190] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:02.460303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d86c1e-4b69-4d76-b8e0-cc39266af6f8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:02.476227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:02.501187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644a475d-6738-460e-88ee-c885d8e2a378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:02.501852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644a475d-6738-460e-88ee-c885d8e2a378 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:02.502136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644a475d-6738-460e-88ee-c885d8e2a378 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:02.502450 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644a475d-6738-460e-88ee-c885d8e2a378 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:02.513877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644a475d-6738-460e-88ee-c885d8e2a378 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-644a475d-6738-460e-88ee-c885d8e2a378 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:02.520185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:02.521043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:02.523300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 100/191] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:02.527398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33cdd8a-ad2b-49e1-ab5a-661b55f01c5c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:02.634855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33cdd8a-ad2b-49e1-ab5a-661b55f01c5c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:02.646560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:02.653229 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644a475d-6738-460e-88ee-c885d8e2a378 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:02.671294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:02.672428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:02.672712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:02.673085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:02.673772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:02.674075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:02.675504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:02.685089 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 92/192] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:02.691545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:02.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644a475d-6738-460e-88ee-c885d8e2a378 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:02.805394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f2d058-8c4b-4b2b-b8d9-3beccd6cdf6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:02.805702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9534bde3-df56-4680-b2e0-824288582471 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:02.805942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f2d058-8c4b-4b2b-b8d9-3beccd6cdf6c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:02.806219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f2d058-8c4b-4b2b-b8d9-3beccd6cdf6c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:02.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f2d058-8c4b-4b2b-b8d9-3beccd6cdf6c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:02.807633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f2d058-8c4b-4b2b-b8d9-3beccd6cdf6c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:02.810116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f2d058-8c4b-4b2b-b8d9-3beccd6cdf6c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:02.811765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 101/193] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:02.817193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f2d058-8c4b-4b2b-b8d9-3beccd6cdf6c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:02.842871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f10916d-976c-4485-aaaa-884d48d8a0c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:02.844268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10916d-976c-4485-aaaa-884d48d8a0c2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:02.844727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10916d-976c-4485-aaaa-884d48d8a0c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:02.845004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10916d-976c-4485-aaaa-884d48d8a0c2 None admin] PATH_INFO: `/v3/users/f9bd29daba4b4b83937dc971049b5999` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:02.859734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f10916d-976c-4485-aaaa-884d48d8a0c2 None admin] RBAC: Authorizing `identity:get_user(user_id=f9bd29daba4b4b83937dc971049b5999)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:02.861332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f10916d-976c-4485-aaaa-884d48d8a0c2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:02.867675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 93/194] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:02 2022] GET /identity/v3/users/f9bd29daba4b4b83937dc971049b5999 => generated 297 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:02.868635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f10916d-976c-4485-aaaa-884d48d8a0c2 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:02.869757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:02.903617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:02.932696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f2d058-8c4b-4b2b-b8d9-3beccd6cdf6c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:02.967754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd581bc-f817-4ebb-a5e3-3d8320884ee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:02.968391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd581bc-f817-4ebb-a5e3-3d8320884ee7 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:02.969158 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd581bc-f817-4ebb-a5e3-3d8320884ee7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:02.969432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd581bc-f817-4ebb-a5e3-3d8320884ee7 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:02.970188 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd581bc-f817-4ebb-a5e3-3d8320884ee7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:02.971686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd581bc-f817-4ebb-a5e3-3d8320884ee7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:02.979055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f10916d-976c-4485-aaaa-884d48d8a0c2 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:03.006523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 102/195] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:27:02 2022] POST /identity/v3/users => generated 293 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:27:03.057577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303ee4ad-8ef0-40f9-8db2-654c2f97ec1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:03.059533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ee4ad-8ef0-40f9-8db2-654c2f97ec1d None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:03.060331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ee4ad-8ef0-40f9-8db2-654c2f97ec1d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:03.060943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ee4ad-8ef0-40f9-8db2-654c2f97ec1d None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:03.064313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303ee4ad-8ef0-40f9-8db2-654c2f97ec1d None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:03.067960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303ee4ad-8ef0-40f9-8db2-654c2f97ec1d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:03.078633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 94/196] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:02 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:03.081032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd581bc-f817-4ebb-a5e3-3d8320884ee7 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:03.206306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:03.212847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd581bc-f817-4ebb-a5e3-3d8320884ee7 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:03.256880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b8a72a-7020-4ae0-9752-666c093a7939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:03.257655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b8a72a-7020-4ae0-9752-666c093a7939 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:03.257976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b8a72a-7020-4ae0-9752-666c093a7939 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:03.258278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b8a72a-7020-4ae0-9752-666c093a7939 None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/users/f9bd29daba4b4b83937dc971049b5999/roles/c65aae75f4b449d0adda4ee067622739` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:03.260529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b8a72a-7020-4ae0-9752-666c093a7939 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, user_id=f9bd29daba4b4b83937dc971049b5999, role_id=c65aae75f4b449d0adda4ee067622739)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:03.262114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b8a72a-7020-4ae0-9752-666c093a7939 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:03.281127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 103/197] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:03 2022] PUT /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/users/f9bd29daba4b4b83937dc971049b5999/roles/c65aae75f4b449d0adda4ee067622739 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:03.402457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d61bba6-4bf0-4eec-b628-d034277a0b6e None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:03.402815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d61bba6-4bf0-4eec-b628-d034277a0b6e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:03.402999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d61bba6-4bf0-4eec-b628-d034277a0b6e None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:03.404731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 95/198] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:03.411968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47269e7e-7ca1-49eb-89f1-b03cff9821fc None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:03.412300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47269e7e-7ca1-49eb-89f1-b03cff9821fc None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:03.412520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47269e7e-7ca1-49eb-89f1-b03cff9821fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:03.433389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-47269e7e-7ca1-49eb-89f1-b03cff9821fc None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:03.434724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-47269e7e-7ca1-49eb-89f1-b03cff9821fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:03.520681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 104/199] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:03.805631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d61bba6-4bf0-4eec-b628-d034277a0b6e None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:03.923522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:03.929942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d61bba6-4bf0-4eec-b628-d034277a0b6e None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:03.970316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b16cb-2b31-41f9-9236-5d1733b63850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:03.971052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b16cb-2b31-41f9-9236-5d1733b63850 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:03.971367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b16cb-2b31-41f9-9236-5d1733b63850 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:03.971695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b16cb-2b31-41f9-9236-5d1733b63850 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:03.984644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b16cb-2b31-41f9-9236-5d1733b63850 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-ef9b16cb-2b31-41f9-9236-5d1733b63850 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:03.990048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:03.990952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:03.991761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 96/200] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:04.001323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47269e7e-7ca1-49eb-89f1-b03cff9821fc None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:04.128111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:04.142299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47269e7e-7ca1-49eb-89f1-b03cff9821fc None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:04.181369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:04.181994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:04.182275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:04.182579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:04.183252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] RBAC: Adding query filter params (name=member) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:04.183554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:04.184939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:04.195885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 105/201] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:04.201829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b16cb-2b31-41f9-9236-5d1733b63850 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:04.317492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b16cb-2b31-41f9-9236-5d1733b63850 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:04.355614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bc3768-7676-4570-a5fa-551dbcecdc9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:04.356250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bc3768-7676-4570-a5fa-551dbcecdc9b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:04.356527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bc3768-7676-4570-a5fa-551dbcecdc9b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:04.356796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bc3768-7676-4570-a5fa-551dbcecdc9b None admin] PATH_INFO: `/v3/users/8e34c04d1dc140318b4b218ac6d6dbcd` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:04.357922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bc3768-7676-4570-a5fa-551dbcecdc9b None admin] RBAC: Authorizing `identity:get_user(user_id=8e34c04d1dc140318b4b218ac6d6dbcd)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:04.360001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bc3768-7676-4570-a5fa-551dbcecdc9b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:04.366259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 97/202] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:04 2022] GET /identity/v3/users/8e34c04d1dc140318b4b218ac6d6dbcd => generated 297 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:04.367658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:04.479301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e523a2-55e9-4194-a6c5-2f62ad55b558 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:04.518247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7721d1-c3b8-4b8a-b92b-84d431f66ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:04.518906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7721d1-c3b8-4b8a-b92b-84d431f66ad1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:04.519185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7721d1-c3b8-4b8a-b92b-84d431f66ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:04.519461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7721d1-c3b8-4b8a-b92b-84d431f66ad1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:04.542910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:04.595198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7721d1-c3b8-4b8a-b92b-84d431f66ad1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:04.596666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7721d1-c3b8-4b8a-b92b-84d431f66ad1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:04.702211 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 106/203] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:04.709053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bc3768-7676-4570-a5fa-551dbcecdc9b None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:04.738643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:04.825252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bc3768-7676-4570-a5fa-551dbcecdc9b None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:04.864873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2361b78d-0991-41c3-b905-85985fefade9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:04.865527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361b78d-0991-41c3-b905-85985fefade9 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:04.865864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361b78d-0991-41c3-b905-85985fefade9 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:04.866142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361b78d-0991-41c3-b905-85985fefade9 None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:04.867413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2361b78d-0991-41c3-b905-85985fefade9 None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:04.868932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2361b78d-0991-41c3-b905-85985fefade9 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:04.870933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 98/204] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:04 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:04.877594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7721d1-c3b8-4b8a-b92b-84d431f66ad1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:04.932712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:04.990470 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7721d1-c3b8-4b8a-b92b-84d431f66ad1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:05.032626 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edb2c91-bb6b-44ae-a5c6-094ac0047162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:05.033301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edb2c91-bb6b-44ae-a5c6-094ac0047162 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:05.033589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edb2c91-bb6b-44ae-a5c6-094ac0047162 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:05.033867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edb2c91-bb6b-44ae-a5c6-094ac0047162 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:05.034732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edb2c91-bb6b-44ae-a5c6-094ac0047162 None admin] RBAC: Adding query filter params (role.id=d729ee72a6d746959af49c5cab72ddd6, scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, user.id=8e34c04d1dc140318b4b218ac6d6dbcd) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:05.035046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edb2c91-bb6b-44ae-a5c6-094ac0047162 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:05.036688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edb2c91-bb6b-44ae-a5c6-094ac0047162 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:05.044469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 107/205] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:04 2022] GET /identity/v3/role_assignments?user.id=8e34c04d1dc140318b4b218ac6d6dbcd&scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca&role.id=d729ee72a6d746959af49c5cab72ddd6 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:05.443898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b8a21d-9ea9-4a72-8956-340ccbbde3f2 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:05.444386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b8a21d-9ea9-4a72-8956-340ccbbde3f2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:05.444386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b8a21d-9ea9-4a72-8956-340ccbbde3f2 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:05.445708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 99/206] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:05.450834 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14eef4-1a50-4dc6-91d6-9a6caca5ecc8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:05.451124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14eef4-1a50-4dc6-91d6-9a6caca5ecc8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:05.451395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b14eef4-1a50-4dc6-91d6-9a6caca5ecc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:05.471222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-3b14eef4-1a50-4dc6-91d6-9a6caca5ecc8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:05.472412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3b14eef4-1a50-4dc6-91d6-9a6caca5ecc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:05.482151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda7607d-5257-4f63-b488-7a079b0a0bb9 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:05.482873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda7607d-5257-4f63-b488-7a079b0a0bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:05.483204 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda7607d-5257-4f63-b488-7a079b0a0bb9 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:05.485132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 100/207] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:05.490730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222bb49-9edb-4948-b133-a3be79371d79 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:05.491012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222bb49-9edb-4948-b133-a3be79371d79 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:05.491285 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6222bb49-9edb-4948-b133-a3be79371d79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:05.509211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6222bb49-9edb-4948-b133-a3be79371d79 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:05.510391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6222bb49-9edb-4948-b133-a3be79371d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:05.542016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 108/208] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:05.590682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 101/209] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:05.681462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b23271-937b-4472-b269-b14836fb79f1 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:05.681790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b23271-937b-4472-b269-b14836fb79f1 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:05.682020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b23271-937b-4472-b269-b14836fb79f1 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:05.683544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 109/210] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:05.689318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6af6ab-6ada-4cc8-b38a-f92fbe32241c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:05.689612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6af6ab-6ada-4cc8-b38a-f92fbe32241c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:05.689887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6af6ab-6ada-4cc8-b38a-f92fbe32241c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:05.708798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-af6af6ab-6ada-4cc8-b38a-f92fbe32241c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:05.710098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-af6af6ab-6ada-4cc8-b38a-f92fbe32241c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:05.729785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:05.782957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 102/211] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:05.809007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b23271-937b-4472-b269-b14836fb79f1 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:05.858425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:05.870056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6af6ab-6ada-4cc8-b38a-f92fbe32241c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:05.928847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b23271-937b-4472-b269-b14836fb79f1 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:05.970366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeef41b0-d645-4893-a627-470d1aa0a948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:05.971178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeef41b0-d645-4893-a627-470d1aa0a948 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:05.971275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeef41b0-d645-4893-a627-470d1aa0a948 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:05.971505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeef41b0-d645-4893-a627-470d1aa0a948 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:05.984335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeef41b0-d645-4893-a627-470d1aa0a948 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-aeef41b0-d645-4893-a627-470d1aa0a948 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:05.989743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:05.990773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:05.991573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 110/212] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:05.996608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6af6ab-6ada-4cc8-b38a-f92fbe32241c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:05.998437 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeef41b0-d645-4893-a627-470d1aa0a948 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:06.036233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f297a9c-94cd-405c-a6ca-1a3a80dbf73f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.036872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f297a9c-94cd-405c-a6ca-1a3a80dbf73f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.037149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f297a9c-94cd-405c-a6ca-1a3a80dbf73f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.037424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f297a9c-94cd-405c-a6ca-1a3a80dbf73f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:06.038546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f297a9c-94cd-405c-a6ca-1a3a80dbf73f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:06.040003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f297a9c-94cd-405c-a6ca-1a3a80dbf73f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:06.041561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 103/213] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:06.046966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f297a9c-94cd-405c-a6ca-1a3a80dbf73f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:06.075181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:06.123035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeef41b0-d645-4893-a627-470d1aa0a948 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:06.127594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:06.153443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f297a9c-94cd-405c-a6ca-1a3a80dbf73f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:06.164600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.165210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.165491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.165761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:06.166421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] RBAC: Adding query filter params (name=member) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:06.166750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:06.168160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:06.177090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 111/214] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:06.177090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:06.189558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa35677-cb11-4a92-be3d-e08b9cc75033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.190169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa35677-cb11-4a92-be3d-e08b9cc75033 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.190446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa35677-cb11-4a92-be3d-e08b9cc75033 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.190745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa35677-cb11-4a92-be3d-e08b9cc75033 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:06.191393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa35677-cb11-4a92-be3d-e08b9cc75033 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:06.192826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa35677-cb11-4a92-be3d-e08b9cc75033 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:06.214303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 104/215] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:27:06 2022] POST /identity/v3/users => generated 293 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:06.215157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa35677-cb11-4a92-be3d-e08b9cc75033 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:06.301546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8413a-93d1-45cf-9347-65c338eb11ab None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:06.309776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:06.338643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa35677-cb11-4a92-be3d-e08b9cc75033 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:06.338888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a378d72f-9a0b-4023-9fec-d24d59c118c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.339364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378d72f-9a0b-4023-9fec-d24d59c118c9 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.339705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378d72f-9a0b-4023-9fec-d24d59c118c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.339980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378d72f-9a0b-4023-9fec-d24d59c118c9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:06.355919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a378d72f-9a0b-4023-9fec-d24d59c118c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-a378d72f-9a0b-4023-9fec-d24d59c118c9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:06.362908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:06.363932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:06.364721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:06.369057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 112/216] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:06.370098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a378d72f-9a0b-4023-9fec-d24d59c118c9 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:06.383556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc4551-c068-4eea-b4a1-1e71c20c1b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.384474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc4551-c068-4eea-b4a1-1e71c20c1b3d None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.384855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc4551-c068-4eea-b4a1-1e71c20c1b3d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.385243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc4551-c068-4eea-b4a1-1e71c20c1b3d None admin] PATH_INFO: `/v3/users/9def1b33260a4402917aa5d19a288cd7` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:06.403522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:06.404786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccc4551-c068-4eea-b4a1-1e71c20c1b3d None admin] RBAC: Authorizing `identity:get_user(user_id=9def1b33260a4402917aa5d19a288cd7)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:06.406430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccc4551-c068-4eea-b4a1-1e71c20c1b3d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:06.408068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 105/217] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:06 2022] GET /identity/v3/users/9def1b33260a4402917aa5d19a288cd7 => generated 293 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:06.416964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc4551-c068-4eea-b4a1-1e71c20c1b3d None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:06.513236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a378d72f-9a0b-4023-9fec-d24d59c118c9 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:06.557556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc4551-c068-4eea-b4a1-1e71c20c1b3d None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:06.574304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.575576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.575962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.576399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:06.577099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:06.577405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:06.579146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:06.590070 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 113/218] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:06.596652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:06.600398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.601130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.601455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.601746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:06.603628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] RBAC: Adding query filter params (role.id=d729ee72a6d746959af49c5cab72ddd6, user.id=9def1b33260a4402917aa5d19a288cd7, scope.system=all) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:06.603628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:06.604594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:06.610969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:06.618430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 106/219] 10.222.0.26 () {56 vars in 1308 bytes} [Wed Oct 19 00:27:06 2022] GET /identity/v3/role_assignments?user.id=9def1b33260a4402917aa5d19a288cd7&scope.system=all&role.id=d729ee72a6d746959af49c5cab72ddd6 => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:06.721768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:06.736081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f920a4-e592-43fa-a034-aaec4de0535f None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:06.784024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c7694c-8cc0-484d-91fe-4d3b61fe91d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.784700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c7694c-8cc0-484d-91fe-4d3b61fe91d7 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.784978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c7694c-8cc0-484d-91fe-4d3b61fe91d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.785252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c7694c-8cc0-484d-91fe-4d3b61fe91d7 None admin] PATH_INFO: `/v3/users/f9bd29daba4b4b83937dc971049b5999` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:06.786766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c7694c-8cc0-484d-91fe-4d3b61fe91d7 None admin] RBAC: Authorizing `identity:get_user(user_id=f9bd29daba4b4b83937dc971049b5999)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:06.788334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c7694c-8cc0-484d-91fe-4d3b61fe91d7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:06.790451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 114/220] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:06 2022] GET /identity/v3/users/f9bd29daba4b4b83937dc971049b5999 => generated 297 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:06.800067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:06.922233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c79191-27db-4dcd-90ef-88aa4702882f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:06.965919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0083aeae-5cff-4dc7-bdfb-69cd7b1abc8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:06.966579 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0083aeae-5cff-4dc7-bdfb-69cd7b1abc8e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:06.966886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0083aeae-5cff-4dc7-bdfb-69cd7b1abc8e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:06.967150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0083aeae-5cff-4dc7-bdfb-69cd7b1abc8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:07.047909 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:07.050971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0083aeae-5cff-4dc7-bdfb-69cd7b1abc8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:07.052913 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0083aeae-5cff-4dc7-bdfb-69cd7b1abc8e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:07.180132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 107/221] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:07.186307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c7694c-8cc0-484d-91fe-4d3b61fe91d7 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:07.318787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c7694c-8cc0-484d-91fe-4d3b61fe91d7 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:07.364104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a206fe1-85f4-477a-aa83-d39dde61bd29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:07.364777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a206fe1-85f4-477a-aa83-d39dde61bd29 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:07.365059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a206fe1-85f4-477a-aa83-d39dde61bd29 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:07.365334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a206fe1-85f4-477a-aa83-d39dde61bd29 None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:07.366611 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a206fe1-85f4-477a-aa83-d39dde61bd29 None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:07.368099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a206fe1-85f4-477a-aa83-d39dde61bd29 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 115/222] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:07 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:07.379833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0083aeae-5cff-4dc7-bdfb-69cd7b1abc8e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:07.505690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:07.508558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0083aeae-5cff-4dc7-bdfb-69cd7b1abc8e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:07.550953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f1019e-7ffc-4d63-b290-6b029fc25941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:07.551585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1019e-7ffc-4d63-b290-6b029fc25941 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:07.551865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1019e-7ffc-4d63-b290-6b029fc25941 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:07.552140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1019e-7ffc-4d63-b290-6b029fc25941 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:07.552883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f1019e-7ffc-4d63-b290-6b029fc25941 None admin] RBAC: Adding query filter params (role.id=c65aae75f4b449d0adda4ee067622739, scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, user.id=f9bd29daba4b4b83937dc971049b5999) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:07.553189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f1019e-7ffc-4d63-b290-6b029fc25941 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:07.554673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f1019e-7ffc-4d63-b290-6b029fc25941 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:07.564407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 108/223] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:07 2022] GET /identity/v3/role_assignments?user.id=f9bd29daba4b4b83937dc971049b5999&scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca&role.id=c65aae75f4b449d0adda4ee067622739 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:08.762771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b829439-c527-4509-a29c-b48a7a8cc943 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:08.763528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b829439-c527-4509-a29c-b48a7a8cc943 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:08.763528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b829439-c527-4509-a29c-b48a7a8cc943 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:08.764939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 116/224] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:27:08.770915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910f92ad-c799-4b30-941e-42ef44eff443 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:08.771387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910f92ad-c799-4b30-941e-42ef44eff443 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:08.771459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910f92ad-c799-4b30-941e-42ef44eff443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:08.791092 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-910f92ad-c799-4b30-941e-42ef44eff443 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:08.792323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-910f92ad-c799-4b30-941e-42ef44eff443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:08.801336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e48cc4-3d0f-4e5a-a56b-9d637abd9d29 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:08.801638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e48cc4-3d0f-4e5a-a56b-9d637abd9d29 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:08.801915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e48cc4-3d0f-4e5a-a56b-9d637abd9d29 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:08.803535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 117/225] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:27:08.808427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b32bb00-d54f-48be-bf94-6b66d14f944c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:08.808735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b32bb00-d54f-48be-bf94-6b66d14f944c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:08.809053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b32bb00-d54f-48be-bf94-6b66d14f944c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:08.830164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8b32bb00-d54f-48be-bf94-6b66d14f944c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:08.831435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8b32bb00-d54f-48be-bf94-6b66d14f944c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:08.854469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:08.869357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 109/226] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:08.910779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 118/227] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:08.961567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24895f43-254b-445d-8ee9-8155c6fa175a None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:08.961858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24895f43-254b-445d-8ee9-8155c6fa175a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:08.962117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24895f43-254b-445d-8ee9-8155c6fa175a None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:08.963618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 110/228] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:08.968639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac7e30f-d37c-4e44-8c1b-169276949b67 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:08.968961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac7e30f-d37c-4e44-8c1b-169276949b67 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:08.969238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac7e30f-d37c-4e44-8c1b-169276949b67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:08.987837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-2ac7e30f-d37c-4e44-8c1b-169276949b67 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:08.989017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2ac7e30f-d37c-4e44-8c1b-169276949b67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:09.074519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 119/229] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:09.161601 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24895f43-254b-445d-8ee9-8155c6fa175a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.189765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac7e30f-d37c-4e44-8c1b-169276949b67 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.288507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:09.298004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24895f43-254b-445d-8ee9-8155c6fa175a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:09.304802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:09.323414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac7e30f-d37c-4e44-8c1b-169276949b67 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:09.334966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690f49ea-d8c4-40c0-a341-b88833f3ad83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:09.335525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690f49ea-d8c4-40c0-a341-b88833f3ad83 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:09.335799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690f49ea-d8c4-40c0-a341-b88833f3ad83 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:09.336067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690f49ea-d8c4-40c0-a341-b88833f3ad83 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:09.346652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690f49ea-d8c4-40c0-a341-b88833f3ad83 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-690f49ea-d8c4-40c0-a341-b88833f3ad83 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:09.351709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:09.352798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:09.353599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 111/230] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:09 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:09.359274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690f49ea-d8c4-40c0-a341-b88833f3ad83 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.371051 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9a00e-8d91-4e4b-996b-0a123e4e5b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:09.371807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9a00e-8d91-4e4b-996b-0a123e4e5b89 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:09.372088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9a00e-8d91-4e4b-996b-0a123e4e5b89 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:09.372359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9a00e-8d91-4e4b-996b-0a123e4e5b89 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:09.373475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b9a00e-8d91-4e4b-996b-0a123e4e5b89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:09.375687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b9a00e-8d91-4e4b-996b-0a123e4e5b89 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:09.378030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 120/231] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:09.380280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9a00e-8d91-4e4b-996b-0a123e4e5b89 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.463937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690f49ea-d8c4-40c0-a341-b88833f3ad83 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:09.486916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9a00e-8d91-4e4b-996b-0a123e4e5b89 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:09.500098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:09.500737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:09.501199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:09.501470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:09.502138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:09.502436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:09.503871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:09.512491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 112/232] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:09 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:09.515993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.524098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0e1de6-4e52-413d-95db-f7e99559ec42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:09.524741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0e1de6-4e52-413d-95db-f7e99559ec42 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:09.525048 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0e1de6-4e52-413d-95db-f7e99559ec42 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:09.525380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0e1de6-4e52-413d-95db-f7e99559ec42 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:09.536005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:09.538213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0e1de6-4e52-413d-95db-f7e99559ec42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-8f0e1de6-4e52-413d-95db-f7e99559ec42 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:09.544499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:09.545451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:09.546247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 121/233] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:09.547832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0e1de6-4e52-413d-95db-f7e99559ec42 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.568286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:09.625759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d392ca-310b-450c-b33b-f348abbc222b None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:09.657430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0e1de6-4e52-413d-95db-f7e99559ec42 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:09.660583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a9301c-3ee0-473e-9b62-60a51a270fa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:09.661196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a9301c-3ee0-473e-9b62-60a51a270fa7 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:09.661471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a9301c-3ee0-473e-9b62-60a51a270fa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:09.661739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a9301c-3ee0-473e-9b62-60a51a270fa7 None admin] PATH_INFO: `/v3/groups` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:09.662361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a9301c-3ee0-473e-9b62-60a51a270fa7 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:09.663842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a9301c-3ee0-473e-9b62-60a51a270fa7 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:09.680049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 113/234] 10.222.0.26 () {60 vars in 1134 bytes} [Wed Oct 19 00:27:09 2022] POST /identity/v3/groups => generated 229 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:09.680049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a9301c-3ee0-473e-9b62-60a51a270fa7 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.697927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd570bea-028c-47ef-8c94-a7763d1e7932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:09.698617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd570bea-028c-47ef-8c94-a7763d1e7932 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:09.698894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd570bea-028c-47ef-8c94-a7763d1e7932 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:09.699161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd570bea-028c-47ef-8c94-a7763d1e7932 None admin] PATH_INFO: `/v3/users/3fc7c367d5094c0f848e5f3ca09da6d4` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:09.717306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd570bea-028c-47ef-8c94-a7763d1e7932 None admin] RBAC: Authorizing `identity:get_user(user_id=3fc7c367d5094c0f848e5f3ca09da6d4)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:09.721411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd570bea-028c-47ef-8c94-a7763d1e7932 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:09.725066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 122/235] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:09 2022] GET /identity/v3/users/3fc7c367d5094c0f848e5f3ca09da6d4 => generated 293 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:09.731246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd570bea-028c-47ef-8c94-a7763d1e7932 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.776356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:09.810943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a9301c-3ee0-473e-9b62-60a51a270fa7 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:09.829996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:09.861145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:09.862104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:09.862576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:09.863054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:09.863934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] RBAC: Adding query filter params (name=member) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:09.864408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:09.866050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:09.869939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd570bea-028c-47ef-8c94-a7763d1e7932 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:09.877558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 114/236] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:09.883824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:09.921199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:09.922624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:09.922911 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:09.923209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:09.924042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] RBAC: Adding query filter params (role.id=c65aae75f4b449d0adda4ee067622739, user.id=3fc7c367d5094c0f848e5f3ca09da6d4, scope.system=all) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:09.924382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:09.926001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:09.931873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 123/237] 10.222.0.26 () {56 vars in 1308 bytes} [Wed Oct 19 00:27:09 2022] GET /identity/v3/role_assignments?user.id=3fc7c367d5094c0f848e5f3ca09da6d4&scope.system=all&role.id=c65aae75f4b449d0adda4ee067622739 => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:10.007659 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a96abb-e633-4c36-8b03-41e41d20aa10 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:10.047303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec98141-7114-4fc6-bb89-598c72199c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:10.047958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec98141-7114-4fc6-bb89-598c72199c14 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:10.048307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec98141-7114-4fc6-bb89-598c72199c14 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:10.048502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec98141-7114-4fc6-bb89-598c72199c14 None admin] PATH_INFO: `/v3/users/9def1b33260a4402917aa5d19a288cd7` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:10.049713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec98141-7114-4fc6-bb89-598c72199c14 None admin] RBAC: Authorizing `identity:get_user(user_id=9def1b33260a4402917aa5d19a288cd7)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:10.051236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec98141-7114-4fc6-bb89-598c72199c14 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:10.053190 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 115/238] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:09 2022] GET /identity/v3/users/9def1b33260a4402917aa5d19a288cd7 => generated 293 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:10.059488 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:10.182600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac54b5fa-48cb-42c6-b4ed-4b78e52d4450 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:10.222690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970632a9-d019-4771-bbd1-cc69a585fa22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:10.223620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970632a9-d019-4771-bbd1-cc69a585fa22 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:10.223901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970632a9-d019-4771-bbd1-cc69a585fa22 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:10.224177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970632a9-d019-4771-bbd1-cc69a585fa22 None admin] PATH_INFO: `/v3/system/users/9def1b33260a4402917aa5d19a288cd7/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:10.225620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970632a9-d019-4771-bbd1-cc69a585fa22 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=9def1b33260a4402917aa5d19a288cd7, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:10.227048 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970632a9-d019-4771-bbd1-cc69a585fa22 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:10.228735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:10.245198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 124/239] 10.222.0.26 () {58 vars in 1334 bytes} [Wed Oct 19 00:27:10 2022] PUT /identity/v3/system/users/9def1b33260a4402917aa5d19a288cd7/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:27:11.919282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a624e4d-41af-4984-a061-ae99c8cad8a3 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:11.919826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a624e4d-41af-4984-a061-ae99c8cad8a3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:11.919826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a624e4d-41af-4984-a061-ae99c8cad8a3 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:11.921103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 116/240] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:11.926504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501e162-00b2-43e9-abe3-34ae9789d935 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:11.926924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501e162-00b2-43e9-abe3-34ae9789d935 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:11.927112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501e162-00b2-43e9-abe3-34ae9789d935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:11.946171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-d501e162-00b2-43e9-abe3-34ae9789d935 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:11.947394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d501e162-00b2-43e9-abe3-34ae9789d935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:12.017499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 125/241] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:12.288441 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a624e4d-41af-4984-a061-ae99c8cad8a3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:12.290609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5e5f4f-068d-4b21-8ddf-e93f4c6a7843 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:12.290902 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5e5f4f-068d-4b21-8ddf-e93f4c6a7843 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:12.291355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5e5f4f-068d-4b21-8ddf-e93f4c6a7843 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:12.292583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 126/242] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:27:12.298259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:12.298729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d6d55-afe7-4de8-b6ad-446d94f20c79 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:12.299024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d6d55-afe7-4de8-b6ad-446d94f20c79 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:12.299297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d6d55-afe7-4de8-b6ad-446d94f20c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:12.318688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1b4d6d55-afe7-4de8-b6ad-446d94f20c79 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:12.319957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1b4d6d55-afe7-4de8-b6ad-446d94f20c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:12.403587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 127/243] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:12.409273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a624e4d-41af-4984-a061-ae99c8cad8a3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:12.438720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b365e-6241-4bf5-9fad-e8a2ae258d9c None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:12.439024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b365e-6241-4bf5-9fad-e8a2ae258d9c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:12.439281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b365e-6241-4bf5-9fad-e8a2ae258d9c None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:12.440752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 128/244] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:27:12.445696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c4046-bec1-4bfb-91c4-d1e420a21a96 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:12.445990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c4046-bec1-4bfb-91c4-d1e420a21a96 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:12.446252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c4046-bec1-4bfb-91c4-d1e420a21a96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:12.447283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1929912-e901-487a-ad04-98dc1b579eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:12.447976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1929912-e901-487a-ad04-98dc1b579eeb None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:12.448300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1929912-e901-487a-ad04-98dc1b579eeb None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:12.448633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1929912-e901-487a-ad04-98dc1b579eeb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:12.449853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1929912-e901-487a-ad04-98dc1b579eeb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:12.451500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1929912-e901-487a-ad04-98dc1b579eeb None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:12.454529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 117/245] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:12.459423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1929912-e901-487a-ad04-98dc1b579eeb None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:12.466074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c82c4046-bec1-4bfb-91c4-d1e420a21a96 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:12.467295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c82c4046-bec1-4bfb-91c4-d1e420a21a96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:12.527767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:12.541359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:12.548705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 129/246] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:12.573918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1929912-e901-487a-ad04-98dc1b579eeb None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:12.609391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446d892a-2a62-495f-beba-c2a67ce63187 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:12.610071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446d892a-2a62-495f-beba-c2a67ce63187 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:12.610356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446d892a-2a62-495f-beba-c2a67ce63187 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:12.610664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446d892a-2a62-495f-beba-c2a67ce63187 None admin] PATH_INFO: `/v3/groups` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:12.611294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446d892a-2a62-495f-beba-c2a67ce63187 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:12.612749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446d892a-2a62-495f-beba-c2a67ce63187 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:12.634517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 118/247] 10.222.0.26 () {60 vars in 1134 bytes} [Wed Oct 19 00:27:12 2022] POST /identity/v3/groups => generated 226 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:27:12.684080 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82c4046-bec1-4bfb-91c4-d1e420a21a96 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:12.802509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82c4046-bec1-4bfb-91c4-d1e420a21a96 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:12.835605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446d892a-2a62-495f-beba-c2a67ce63187 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:12.845498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40afc7d-ebf2-4730-bfbc-0b566ece37a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:12.846138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40afc7d-ebf2-4730-bfbc-0b566ece37a4 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:12.846419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40afc7d-ebf2-4730-bfbc-0b566ece37a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:12.846745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40afc7d-ebf2-4730-bfbc-0b566ece37a4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:12.860413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40afc7d-ebf2-4730-bfbc-0b566ece37a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-d40afc7d-ebf2-4730-bfbc-0b566ece37a4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:12.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:12.866587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:12.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 130/248] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:12.872634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40afc7d-ebf2-4730-bfbc-0b566ece37a4 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:12.929615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:12.978334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446d892a-2a62-495f-beba-c2a67ce63187 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:12.996610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40afc7d-ebf2-4730-bfbc-0b566ece37a4 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:13.019483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fab8f1-a899-4126-a5a9-c7e86ebfe018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:13.020293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fab8f1-a899-4126-a5a9-c7e86ebfe018 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:13.020661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fab8f1-a899-4126-a5a9-c7e86ebfe018 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:13.021034 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fab8f1-a899-4126-a5a9-c7e86ebfe018 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:13.023460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:13.036218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fab8f1-a899-4126-a5a9-c7e86ebfe018 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-55fab8f1-a899-4126-a5a9-c7e86ebfe018 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:13.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:13.043521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:13.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 119/249] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:13.049755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fab8f1-a899-4126-a5a9-c7e86ebfe018 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:13.061756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:13.062846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:13.063024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:13.063297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:13.064227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:13.064737 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:13.066525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:13.080889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 131/250] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:13.086414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:13.181281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fab8f1-a899-4126-a5a9-c7e86ebfe018 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:13.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594dfe98-08af-4dd2-bf0d-0b1777f8ae13 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:13.220298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:13.231499 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:13.232286 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:13.232575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:13.232853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:13.233538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] RBAC: Adding query filter params (name=member) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:13.233893 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:13.235505 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:13.246223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 120/251] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:13.249974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1811249-f42e-452a-8c16-c713771abea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:13.251711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1811249-f42e-452a-8c16-c713771abea8 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:13.251711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1811249-f42e-452a-8c16-c713771abea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:13.251922 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1811249-f42e-452a-8c16-c713771abea8 None admin] PATH_INFO: `/v3/users/3fc7c367d5094c0f848e5f3ca09da6d4` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:13.253169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:13.254328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1811249-f42e-452a-8c16-c713771abea8 None admin] RBAC: Authorizing `identity:get_user(user_id=3fc7c367d5094c0f848e5f3ca09da6d4)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:13.255876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1811249-f42e-452a-8c16-c713771abea8 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:13.257724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 132/252] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:13 2022] GET /identity/v3/users/3fc7c367d5094c0f848e5f3ca09da6d4 => generated 293 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:13.263520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1811249-f42e-452a-8c16-c713771abea8 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:13.337869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:13.400231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478d64db-df56-4349-9ace-2503797e2c6f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:13.411894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1811249-f42e-452a-8c16-c713771abea8 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:13.443248 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fd21a8-016d-44aa-8146-627cc6cd8692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:13.444399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fd21a8-016d-44aa-8146-627cc6cd8692 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:13.444978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fd21a8-016d-44aa-8146-627cc6cd8692 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:13.445432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fd21a8-016d-44aa-8146-627cc6cd8692 None admin] PATH_INFO: `/v3/users/9def1b33260a4402917aa5d19a288cd7` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:13.447689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fd21a8-016d-44aa-8146-627cc6cd8692 None admin] RBAC: Authorizing `identity:get_user(user_id=9def1b33260a4402917aa5d19a288cd7)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:13.449796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fd21a8-016d-44aa-8146-627cc6cd8692 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:13.452952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 121/253] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:13 2022] GET /identity/v3/users/9def1b33260a4402917aa5d19a288cd7 => generated 293 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:13.461141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fd21a8-016d-44aa-8146-627cc6cd8692 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:13.470907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f57887-365f-40c6-99f6-f12d64c1d183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:13.472069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f57887-365f-40c6-99f6-f12d64c1d183 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:13.472560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f57887-365f-40c6-99f6-f12d64c1d183 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:13.473297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f57887-365f-40c6-99f6-f12d64c1d183 None admin] PATH_INFO: `/v3/system/users/3fc7c367d5094c0f848e5f3ca09da6d4/roles/c65aae75f4b449d0adda4ee067622739` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:13.478033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f57887-365f-40c6-99f6-f12d64c1d183 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=3fc7c367d5094c0f848e5f3ca09da6d4, role_id=c65aae75f4b449d0adda4ee067622739)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:13.479203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f57887-365f-40c6-99f6-f12d64c1d183 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:13.493310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 133/254] 10.222.0.26 () {58 vars in 1334 bytes} [Wed Oct 19 00:27:13 2022] PUT /identity/v3/system/users/3fc7c367d5094c0f848e5f3ca09da6d4/roles/c65aae75f4b449d0adda4ee067622739 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:27:13.606702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fd21a8-016d-44aa-8146-627cc6cd8692 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:13.637272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:13.653543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d681dad-0bbc-4bd5-9db2-82a64cda07ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:13.654175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d681dad-0bbc-4bd5-9db2-82a64cda07ae None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:13.654458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d681dad-0bbc-4bd5-9db2-82a64cda07ae None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:13.654761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d681dad-0bbc-4bd5-9db2-82a64cda07ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:13.655498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d681dad-0bbc-4bd5-9db2-82a64cda07ae None admin] RBAC: Adding query filter params (role.id=d729ee72a6d746959af49c5cab72ddd6, user.id=9def1b33260a4402917aa5d19a288cd7, scope.system=all) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:13.656055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d681dad-0bbc-4bd5-9db2-82a64cda07ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:13.657679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d681dad-0bbc-4bd5-9db2-82a64cda07ae None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:13.672632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 122/255] 10.222.0.26 () {56 vars in 1308 bytes} [Wed Oct 19 00:27:13 2022] GET /identity/v3/role_assignments?user.id=9def1b33260a4402917aa5d19a288cd7&scope.system=all&role.id=d729ee72a6d746959af49c5cab72ddd6 => generated 510 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:15.336123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f4e073-3444-4194-944f-5475f998e64c None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:15.337170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f4e073-3444-4194-944f-5475f998e64c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:15.337650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f4e073-3444-4194-944f-5475f998e64c None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:15.339532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 134/256] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:15.345186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddbbbef-3280-483c-9787-979dd30aca8e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:15.346019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddbbbef-3280-483c-9787-979dd30aca8e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:15.346518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddbbbef-3280-483c-9787-979dd30aca8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:15.373551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5ddbbbef-3280-483c-9787-979dd30aca8e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:15.376073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5ddbbbef-3280-483c-9787-979dd30aca8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:15.451892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 123/257] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:15 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 00:27:15.862855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fc88e1-b042-4518-8bf6-2e20f6d7ac02 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:15.863598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fc88e1-b042-4518-8bf6-2e20f6d7ac02 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:15.864088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fc88e1-b042-4518-8bf6-2e20f6d7ac02 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:15.866088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 135/258] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:15.872532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5405351c-c56c-4415-bb63-1fd6b867c8b9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:15.872820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5405351c-c56c-4415-bb63-1fd6b867c8b9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:15.873266 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5405351c-c56c-4415-bb63-1fd6b867c8b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:15.900221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5405351c-c56c-4415-bb63-1fd6b867c8b9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:15.901719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5405351c-c56c-4415-bb63-1fd6b867c8b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:15.913837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 124/259] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:15 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 00:27:15.971820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fc88e1-b042-4518-8bf6-2e20f6d7ac02 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:15.988697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fc88e1-b042-4518-8bf6-2e20f6d7ac02 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:15.993594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bb748b-e539-462b-b162-16ab85daa93d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:15.994598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb748b-e539-462b-b162-16ab85daa93d admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:15.995086 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb748b-e539-462b-b162-16ab85daa93d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:15.995561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb748b-e539-462b-b162-16ab85daa93d admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.016024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bb748b-e539-462b-b162-16ab85daa93d admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-d5bb748b-e539-462b-b162-16ab85daa93d admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:16.021766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:16.022775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:16.023668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:16.026115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 136/260] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Oct 19 00:27:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 55 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:16.033046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5405351c-c56c-4415-bb63-1fd6b867c8b9 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.045171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d02bd66-5bfa-4066-880d-b18552e35de0 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.046140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d02bd66-5bfa-4066-880d-b18552e35de0 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.047191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d02bd66-5bfa-4066-880d-b18552e35de0 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.048555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5405351c-c56c-4415-bb63-1fd6b867c8b9 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.049630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 137/261] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:27:16.056406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.056769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb3f193-5f5a-454c-b026-90764ffe455c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.057104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb3f193-5f5a-454c-b026-90764ffe455c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.057416 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb3f193-5f5a-454c-b026-90764ffe455c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.058004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.058517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.058989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.059885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:16.060386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.063223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:27:16.063223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:27:16.063223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:27:16.064527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.087178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 125/262] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:16.088315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-acb3f193-5f5a-454c-b026-90764ffe455c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:16.089842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-acb3f193-5f5a-454c-b026-90764ffe455c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:16.094197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.110333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd23629-d1a4-4629-abe6-51c42c205fb6 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.113630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 138/263] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:16 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 19 00:27:16.121839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.122516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.122807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.123086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] PATH_INFO: `/v3/groups/90d2140ee6e84bbfb742def6466833e8` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.130647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] RBAC: Authorizing `identity:get_group(group_id=90d2140ee6e84bbfb742def6466833e8)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.131793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:27:16.131793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:27:16.131793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:27:16.133017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.135036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 126/264] 10.222.0.26 () {56 vars in 1201 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/groups/90d2140ee6e84bbfb742def6466833e8 => generated 226 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:16.141341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb3f193-5f5a-454c-b026-90764ffe455c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.156720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb3f193-5f5a-454c-b026-90764ffe455c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.161647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f37b7d-2803-4e3a-a8af-4f5508afb29e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.162322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f37b7d-2803-4e3a-a8af-4f5508afb29e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.162672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f37b7d-2803-4e3a-a8af-4f5508afb29e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.162979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f37b7d-2803-4e3a-a8af-4f5508afb29e admin admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.164153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f37b7d-2803-4e3a-a8af-4f5508afb29e admin admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.166541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f37b7d-2803-4e3a-a8af-4f5508afb29e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.168715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 139/265] 10.222.0.26 () {56 vars in 1207 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:16.174709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.185460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8188594-ed69-4bf6-8029-3af27783d123 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.190082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.190814 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.191399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/groups/90d2140ee6e84bbfb742def6466833e8/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.193433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362, group_id=90d2140ee6e84bbfb742def6466833e8, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.198619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:27:16.198619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:27:16.198619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:27:16.200686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.227671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 127/266] 10.222.0.26 () {58 vars in 1463 bytes} [Wed Oct 19 00:27:16 2022] PUT /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/groups/90d2140ee6e84bbfb742def6466833e8/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:16.379840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f37b7d-2803-4e3a-a8af-4f5508afb29e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.395742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f37b7d-2803-4e3a-a8af-4f5508afb29e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.423413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:16.483125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf9ac7-9ff3-4405-afeb-afa6358a23c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.484102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf9ac7-9ff3-4405-afeb-afa6358a23c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.484664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf9ac7-9ff3-4405-afeb-afa6358a23c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.484754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf9ac7-9ff3-4405-afeb-afa6358a23c3 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.501708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bf9ac7-9ff3-4405-afeb-afa6358a23c3 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-22bf9ac7-9ff3-4405-afeb-afa6358a23c3 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:16.506992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:16.507875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:16.508752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 140/267] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 129 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:16.514611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.527395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4435c606-a602-4dd8-9f8d-f3ad08e56162 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.532620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.533284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.533685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.533993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.539148 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:16.539492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.541382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.553085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 128/268] 10.222.0.26 () {56 vars in 1132 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:16.564036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf9ac7-9ff3-4405-afeb-afa6358a23c3 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.575978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf9ac7-9ff3-4405-afeb-afa6358a23c3 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.585623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.586348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.586667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.586942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] PATH_INFO: `/v3/groups/90d2140ee6e84bbfb742def6466833e8` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.588113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] RBAC: Authorizing `identity:get_group(group_id=90d2140ee6e84bbfb742def6466833e8)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.590031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING py.warnings [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:27:16.590031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: warnings.warn(msg) Oct 19 00:27:16.590031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: Oct 19 00:27:16.591420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.594134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 141/269] 10.222.0.26 () {56 vars in 1201 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/groups/90d2140ee6e84bbfb742def6466833e8 => generated 226 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:16.600322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.612224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc59aaf-feee-49bd-a4d7-55ef09caccb4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.621330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7345a62-3df0-4b8a-943f-1a8c7890ab38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.621994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7345a62-3df0-4b8a-943f-1a8c7890ab38 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.622275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7345a62-3df0-4b8a-943f-1a8c7890ab38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.622593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7345a62-3df0-4b8a-943f-1a8c7890ab38 admin admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.623779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7345a62-3df0-4b8a-943f-1a8c7890ab38 admin admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.625969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7345a62-3df0-4b8a-943f-1a8c7890ab38 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.627720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 129/270] 10.222.0.26 () {56 vars in 1207 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:16.633936 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.645460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d2bf15-ef97-4ba0-bb07-a30f6981a585 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.650238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.651008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.651311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.651588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/groups/90d2140ee6e84bbfb742def6466833e8/roles/66029780d52042b9a44109eb91c0d80e` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.653786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362, group_id=90d2140ee6e84bbfb742def6466833e8, role_id=66029780d52042b9a44109eb91c0d80e)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.657573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING py.warnings [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:27:16.657573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: warnings.warn(msg) Oct 19 00:27:16.657573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: Oct 19 00:27:16.659879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.674548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3761f80-ed73-4e30-b810-653f8995201e None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.674876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3761f80-ed73-4e30-b810-653f8995201e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.675208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3761f80-ed73-4e30-b810-653f8995201e None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.676939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 130/271] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:16.680652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 142/272] 10.222.0.26 () {58 vars in 1463 bytes} [Wed Oct 19 00:27:16 2022] PUT /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362/groups/90d2140ee6e84bbfb742def6466833e8/roles/66029780d52042b9a44109eb91c0d80e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:16.682711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2ecb2-763b-4a73-9cc0-d325a3bac03a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.683019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2ecb2-763b-4a73-9cc0-d325a3bac03a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.683332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2ecb2-763b-4a73-9cc0-d325a3bac03a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.704520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-43f2ecb2-763b-4a73-9cc0-d325a3bac03a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:16.705822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-43f2ecb2-763b-4a73-9cc0-d325a3bac03a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:16.709380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.861833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae7828d-f749-4acf-8c13-70933134da8a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.872365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba1a676-41cc-4e09-b455-03d71c32f558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.873042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba1a676-41cc-4e09-b455-03d71c32f558 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.873360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba1a676-41cc-4e09-b455-03d71c32f558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.873640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba1a676-41cc-4e09-b455-03d71c32f558 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.896327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 131/273] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:16.899803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba1a676-41cc-4e09-b455-03d71c32f558 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-4ba1a676-41cc-4e09-b455-03d71c32f558 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:16.905365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:16.906963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:16.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 143/274] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:16.915377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f2ecb2-763b-4a73-9cc0-d325a3bac03a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.935575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f2ecb2-763b-4a73-9cc0-d325a3bac03a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.943092 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.944477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.944947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.945811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:16.946327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.948178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.951390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:16.958874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 132/275] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:16.964713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba1a676-41cc-4e09-b455-03d71c32f558 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:16.981242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba1a676-41cc-4e09-b455-03d71c32f558 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:16.986204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80036219-ea1b-4a6a-b6a4-2550eb5727ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:16.986877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80036219-ea1b-4a6a-b6a4-2550eb5727ac admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:16.987164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80036219-ea1b-4a6a-b6a4-2550eb5727ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:16.987461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80036219-ea1b-4a6a-b6a4-2550eb5727ac admin admin] PATH_INFO: `/v3/groups/90d2140ee6e84bbfb742def6466833e8` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:16.988704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80036219-ea1b-4a6a-b6a4-2550eb5727ac admin admin] RBAC: Authorizing `identity:get_group(group_id=90d2140ee6e84bbfb742def6466833e8)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:16.994867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80036219-ea1b-4a6a-b6a4-2550eb5727ac admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:16.996627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 144/276] 10.222.0.26 () {56 vars in 1201 bytes} [Wed Oct 19 00:27:16 2022] GET /identity/v3/groups/90d2140ee6e84bbfb742def6466833e8 => generated 226 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:17.002930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:17.017447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2e96ac-4ced-4858-acb1-f027d09c61f8 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:17.024987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f6ef37-c8f7-4009-bc4f-c46a9ab2020d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:17.025640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6ef37-c8f7-4009-bc4f-c46a9ab2020d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:17.025905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6ef37-c8f7-4009-bc4f-c46a9ab2020d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:17.026177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6ef37-c8f7-4009-bc4f-c46a9ab2020d admin admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:17.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f6ef37-c8f7-4009-bc4f-c46a9ab2020d admin admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:17.033965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f6ef37-c8f7-4009-bc4f-c46a9ab2020d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:17.035988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 133/277] 10.222.0.26 () {56 vars in 1207 bytes} [Wed Oct 19 00:27:17 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:17.041780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80036219-ea1b-4a6a-b6a4-2550eb5727ac admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:17.053071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80036219-ea1b-4a6a-b6a4-2550eb5727ac admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:17.066262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e55f84-da57-44e5-a7d3-02897b99ab5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:17.067012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e55f84-da57-44e5-a7d3-02897b99ab5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:17.067303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e55f84-da57-44e5-a7d3-02897b99ab5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:17.067579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e55f84-da57-44e5-a7d3-02897b99ab5f admin admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/groups/90d2140ee6e84bbfb742def6466833e8/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:17.069611 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e55f84-da57-44e5-a7d3-02897b99ab5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, group_id=90d2140ee6e84bbfb742def6466833e8, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:17.072391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e55f84-da57-44e5-a7d3-02897b99ab5f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:17.089237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 145/278] 10.222.0.26 () {58 vars in 1463 bytes} [Wed Oct 19 00:27:17 2022] PUT /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/groups/90d2140ee6e84bbfb742def6466833e8/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:17.396241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f6ef37-c8f7-4009-bc4f-c46a9ab2020d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:17.535908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f6ef37-c8f7-4009-bc4f-c46a9ab2020d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:17.581949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a08411-60c0-4b36-bc1e-419f1a3c6df6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:17.582648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a08411-60c0-4b36-bc1e-419f1a3c6df6 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:17.582957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a08411-60c0-4b36-bc1e-419f1a3c6df6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:17.583240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a08411-60c0-4b36-bc1e-419f1a3c6df6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:17.595342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a08411-60c0-4b36-bc1e-419f1a3c6df6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-f2a08411-60c0-4b36-bc1e-419f1a3c6df6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:17.606998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:17.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:17.609393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 134/279] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:17.616486 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e55f84-da57-44e5-a7d3-02897b99ab5f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:17.709732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:17.755125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e55f84-da57-44e5-a7d3-02897b99ab5f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:17.796864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:17.797520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:17.797826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:17.798141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:17.798838 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:17.799182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:17.800637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:17.810197 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 146/280] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:17.816456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a08411-60c0-4b36-bc1e-419f1a3c6df6 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:17.891975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:17.949610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a08411-60c0-4b36-bc1e-419f1a3c6df6 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:17.991956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3ca589-a2e2-48dc-b492-84c7174ba26f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:17.992570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3ca589-a2e2-48dc-b492-84c7174ba26f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:17.992847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3ca589-a2e2-48dc-b492-84c7174ba26f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:17.993140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3ca589-a2e2-48dc-b492-84c7174ba26f None admin] PATH_INFO: `/v3/users/3fc7c367d5094c0f848e5f3ca09da6d4` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:17.994507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3ca589-a2e2-48dc-b492-84c7174ba26f None admin] RBAC: Authorizing `identity:get_user(user_id=3fc7c367d5094c0f848e5f3ca09da6d4)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:17.996140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3ca589-a2e2-48dc-b492-84c7174ba26f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:17.998187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 135/281] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:27:17 2022] GET /identity/v3/users/3fc7c367d5094c0f848e5f3ca09da6d4 => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:18.004549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:18.126527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440910e9-f242-4ef4-a6dd-5ff5024cc1f6 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:18.183540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:18.193505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2465b6de-d5df-46d9-a2eb-8d63c7b30ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:18.194129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2465b6de-d5df-46d9-a2eb-8d63c7b30ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:18.198888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2465b6de-d5df-46d9-a2eb-8d63c7b30ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:18.199182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2465b6de-d5df-46d9-a2eb-8d63c7b30ac4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:18.199926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2465b6de-d5df-46d9-a2eb-8d63c7b30ac4 None admin] RBAC: Adding query filter params (role.id=c65aae75f4b449d0adda4ee067622739, user.id=3fc7c367d5094c0f848e5f3ca09da6d4, scope.system=all) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:18.200237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2465b6de-d5df-46d9-a2eb-8d63c7b30ac4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:18.201696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2465b6de-d5df-46d9-a2eb-8d63c7b30ac4 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:18.215236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802fe79e-1bd2-4896-9d21-354f9615e61a None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:18.215621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802fe79e-1bd2-4896-9d21-354f9615e61a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:18.216119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802fe79e-1bd2-4896-9d21-354f9615e61a None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:18.217690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 136/282] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:18.218071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 147/283] 10.222.0.26 () {56 vars in 1308 bytes} [Wed Oct 19 00:27:18 2022] GET /identity/v3/role_assignments?user.id=3fc7c367d5094c0f848e5f3ca09da6d4&scope.system=all&role.id=c65aae75f4b449d0adda4ee067622739 => generated 510 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:18.243489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e2f37-1b5e-4868-b357-a6a5592af935 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:18.243740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e2f37-1b5e-4868-b357-a6a5592af935 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:18.244100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e2f37-1b5e-4868-b357-a6a5592af935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:18.287703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-be9e2f37-1b5e-4868-b357-a6a5592af935 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:18.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-be9e2f37-1b5e-4868-b357-a6a5592af935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:18.404341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 137/284] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:18 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 19 00:27:18.971629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ad0e3-d9ec-446a-b032-8247b896f336 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:18.972420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ad0e3-d9ec-446a-b032-8247b896f336 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:18.972420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ad0e3-d9ec-446a-b032-8247b896f336 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:18.974274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 148/285] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:18.979657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81fe3f0-d0c6-4bcf-b0c3-aca048fb4651 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:18.980336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81fe3f0-d0c6-4bcf-b0c3-aca048fb4651 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:18.980336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81fe3f0-d0c6-4bcf-b0c3-aca048fb4651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:18.992939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3ad0e3-d9ec-446a-b032-8247b896f336 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.016822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3ad0e3-d9ec-446a-b032-8247b896f336 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.019347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e81fe3f0-d0c6-4bcf-b0c3-aca048fb4651 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:19.020568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e81fe3f0-d0c6-4bcf-b0c3-aca048fb4651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:19.025013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17edda8-37a8-41a3-8cc5-45d5154df891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.025653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17edda8-37a8-41a3-8cc5-45d5154df891 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.025934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17edda8-37a8-41a3-8cc5-45d5154df891 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.026191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17edda8-37a8-41a3-8cc5-45d5154df891 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.040905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 138/286] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:18 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 00:27:19.050357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17edda8-37a8-41a3-8cc5-45d5154df891 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-b17edda8-37a8-41a3-8cc5-45d5154df891 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:19.055393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:19.058721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:19.062361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 149/287] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Oct 19 00:27:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 64 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:19.064927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81fe3f0-d0c6-4bcf-b0c3-aca048fb4651 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.076819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81fe3f0-d0c6-4bcf-b0c3-aca048fb4651 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.085729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.086374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.086822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.087138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.087877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:19.088222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:19.089947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:19.104522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 139/288] 10.222.0.26 () {56 vars in 1132 bytes} [Wed Oct 19 00:27:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:19.110992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17edda8-37a8-41a3-8cc5-45d5154df891 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.130386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17edda8-37a8-41a3-8cc5-45d5154df891 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.138150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c89ec7-9f6b-40b2-9a79-afca71bcb94a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.138831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89ec7-9f6b-40b2-9a79-afca71bcb94a admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.139109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89ec7-9f6b-40b2-9a79-afca71bcb94a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.139428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89ec7-9f6b-40b2-9a79-afca71bcb94a admin admin] PATH_INFO: `/v3/groups/90d2140ee6e84bbfb742def6466833e8` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.143180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c89ec7-9f6b-40b2-9a79-afca71bcb94a admin admin] RBAC: Authorizing `identity:get_group(group_id=90d2140ee6e84bbfb742def6466833e8)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:19.145116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c89ec7-9f6b-40b2-9a79-afca71bcb94a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:19.147016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 150/289] 10.222.0.26 () {56 vars in 1201 bytes} [Wed Oct 19 00:27:19 2022] GET /identity/v3/groups/90d2140ee6e84bbfb742def6466833e8 => generated 226 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:19.152525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.166433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0c2d53-5ea5-409c-8625-429157a6f413 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.174004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8350db-df57-4ed0-99c9-488a1c53faaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.174782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8350db-df57-4ed0-99c9-488a1c53faaa admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.175126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8350db-df57-4ed0-99c9-488a1c53faaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.175359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8350db-df57-4ed0-99c9-488a1c53faaa admin admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.176515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8350db-df57-4ed0-99c9-488a1c53faaa admin admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:19.178843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8350db-df57-4ed0-99c9-488a1c53faaa admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:19.180647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 140/290] 10.222.0.26 () {56 vars in 1207 bytes} [Wed Oct 19 00:27:19 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:19.186403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c89ec7-9f6b-40b2-9a79-afca71bcb94a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.199966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c89ec7-9f6b-40b2-9a79-afca71bcb94a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.210525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10431b4-6426-4116-8616-d69a933a4a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.211226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10431b4-6426-4116-8616-d69a933a4a90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.211503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10431b4-6426-4116-8616-d69a933a4a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.211844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10431b4-6426-4116-8616-d69a933a4a90 admin admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/groups/90d2140ee6e84bbfb742def6466833e8/roles/66029780d52042b9a44109eb91c0d80e` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.219042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10431b4-6426-4116-8616-d69a933a4a90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca, group_id=90d2140ee6e84bbfb742def6466833e8, role_id=66029780d52042b9a44109eb91c0d80e)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:19.221841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10431b4-6426-4116-8616-d69a933a4a90 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:19.241670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 151/291] 10.222.0.26 () {58 vars in 1463 bytes} [Wed Oct 19 00:27:19 2022] PUT /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca/groups/90d2140ee6e84bbfb742def6466833e8/roles/66029780d52042b9a44109eb91c0d80e => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:19.622825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8350db-df57-4ed0-99c9-488a1c53faaa admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.634197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8350db-df57-4ed0-99c9-488a1c53faaa admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.697282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:19.708786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b589f00-f80d-4f8d-a234-45413d9b5f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.709507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b589f00-f80d-4f8d-a234-45413d9b5f3e admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.709808 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b589f00-f80d-4f8d-a234-45413d9b5f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.710084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b589f00-f80d-4f8d-a234-45413d9b5f3e admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.723864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b589f00-f80d-4f8d-a234-45413d9b5f3e admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-9b589f00-f80d-4f8d-a234-45413d9b5f3e admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:19.729773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:19.731929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:19.734353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 141/292] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Oct 19 00:27:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 110 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:19.738201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10431b4-6426-4116-8616-d69a933a4a90 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.755100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10431b4-6426-4116-8616-d69a933a4a90 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.765314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.765939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.766225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.766557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.767246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:19.767584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:19.768583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING py.warnings [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:27:19.768583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: warnings.warn(msg) Oct 19 00:27:19.768583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: Oct 19 00:27:19.769498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:19.794511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 152/293] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Oct 19 00:27:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:19.800593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b589f00-f80d-4f8d-a234-45413d9b5f3e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.812685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b589f00-f80d-4f8d-a234-45413d9b5f3e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.820207 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e280f69-4df6-4bbe-a655-cd67963224ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.820936 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e280f69-4df6-4bbe-a655-cd67963224ff admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.821268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e280f69-4df6-4bbe-a655-cd67963224ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.821593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e280f69-4df6-4bbe-a655-cd67963224ff admin admin] PATH_INFO: `/v3/groups/052736fb793e40ab86a6719651d30372` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.827819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e280f69-4df6-4bbe-a655-cd67963224ff admin admin] RBAC: Authorizing `identity:get_group(group_id=052736fb793e40ab86a6719651d30372)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:19.829767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e280f69-4df6-4bbe-a655-cd67963224ff admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:19.831932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 142/294] 10.222.0.26 () {56 vars in 1201 bytes} [Wed Oct 19 00:27:19 2022] GET /identity/v3/groups/052736fb793e40ab86a6719651d30372 => generated 229 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:19.837157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.853088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50acb54-b769-4a0c-ae7e-a1ad8159169a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.860546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafcd887-2582-4ecd-91c7-e683fe14abb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.861263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcd887-2582-4ecd-91c7-e683fe14abb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.861562 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcd887-2582-4ecd-91c7-e683fe14abb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.861867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcd887-2582-4ecd-91c7-e683fe14abb7 admin admin] PATH_INFO: `/v3/projects/d417ad1948a04e309383739147354388` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.863155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafcd887-2582-4ecd-91c7-e683fe14abb7 admin admin] RBAC: Authorizing `identity:get_project(project_id=d417ad1948a04e309383739147354388)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:19.865193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafcd887-2582-4ecd-91c7-e683fe14abb7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:19.871434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 153/295] 10.222.0.26 () {56 vars in 1207 bytes} [Wed Oct 19 00:27:19 2022] GET /identity/v3/projects/d417ad1948a04e309383739147354388 => generated 344 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:19.877957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e280f69-4df6-4bbe-a655-cd67963224ff admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:19.890681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e280f69-4df6-4bbe-a655-cd67963224ff admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:19.898232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46060cbd-8d47-4855-8cd5-4c83fb02798a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:19.898957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46060cbd-8d47-4855-8cd5-4c83fb02798a admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:19.899239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46060cbd-8d47-4855-8cd5-4c83fb02798a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:19.899544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46060cbd-8d47-4855-8cd5-4c83fb02798a admin admin] PATH_INFO: `/v3/projects/d417ad1948a04e309383739147354388/groups/052736fb793e40ab86a6719651d30372/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:19.901621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46060cbd-8d47-4855-8cd5-4c83fb02798a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d417ad1948a04e309383739147354388, group_id=052736fb793e40ab86a6719651d30372, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:19.904524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46060cbd-8d47-4855-8cd5-4c83fb02798a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:19.928555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 143/296] 10.222.0.26 () {58 vars in 1463 bytes} [Wed Oct 19 00:27:19 2022] PUT /identity/v3/projects/d417ad1948a04e309383739147354388/groups/052736fb793e40ab86a6719651d30372/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:20.153100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fde97c-08ba-4a3f-b2a3-10186698f1c8 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:20.153767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fde97c-08ba-4a3f-b2a3-10186698f1c8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:20.153767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fde97c-08ba-4a3f-b2a3-10186698f1c8 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:20.156218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 154/297] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:20.160931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f663f2-052b-43ab-bbb1-720f3d6073ea None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:20.161527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f663f2-052b-43ab-bbb1-720f3d6073ea None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:20.161527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f663f2-052b-43ab-bbb1-720f3d6073ea None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:20.161852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c1a1a-c5e9-480f-882a-677c9515c4a2 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:20.162040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c1a1a-c5e9-480f-882a-677c9515c4a2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:20.162318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c1a1a-c5e9-480f-882a-677c9515c4a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:20.163325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 144/298] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:20.185581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c74c1a1a-c5e9-480f-882a-677c9515c4a2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:20.186378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab1187d-e011-4c40-85f3-24c35523e2f6 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:20.187386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab1187d-e011-4c40-85f3-24c35523e2f6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:20.187553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c74c1a1a-c5e9-480f-882a-677c9515c4a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:20.187752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab1187d-e011-4c40-85f3-24c35523e2f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:20.217805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-1ab1187d-e011-4c40-85f3-24c35523e2f6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:20.219771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1ab1187d-e011-4c40-85f3-24c35523e2f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:20.308555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 155/299] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:20.312822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 145/300] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:20.627706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74c1a1a-c5e9-480f-882a-677c9515c4a2 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:20.728522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcc812-c78e-463f-875d-f80ed2376bbe None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:20.728848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcc812-c78e-463f-875d-f80ed2376bbe None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:20.729069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfcc812-c78e-463f-875d-f80ed2376bbe None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:20.731507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:20.731919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 146/301] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:20.736633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debcd925-cf3a-43e8-a3e8-8c709e025122 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:20.736945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debcd925-cf3a-43e8-a3e8-8c709e025122 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:20.737188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debcd925-cf3a-43e8-a3e8-8c709e025122 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:20.761355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-debcd925-cf3a-43e8-a3e8-8c709e025122 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:20.762585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-debcd925-cf3a-43e8-a3e8-8c709e025122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:20.763946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74c1a1a-c5e9-480f-882a-677c9515c4a2 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:20.801248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a1721c-59ca-4cec-905e-1c5d32cd22c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:20.802162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a1721c-59ca-4cec-905e-1c5d32cd22c2 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:20.802567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a1721c-59ca-4cec-905e-1c5d32cd22c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:20.802844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a1721c-59ca-4cec-905e-1c5d32cd22c2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:20.816790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a1721c-59ca-4cec-905e-1c5d32cd22c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-15a1721c-59ca-4cec-905e-1c5d32cd22c2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:20.823311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:20.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:20.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:20.825607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 156/302] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:20.826168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a1721c-59ca-4cec-905e-1c5d32cd22c2 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:20.847310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 147/303] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:20.848577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debcd925-cf3a-43e8-a3e8-8c709e025122 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:20.910985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:21.042754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a1721c-59ca-4cec-905e-1c5d32cd22c2 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:21.055620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debcd925-cf3a-43e8-a3e8-8c709e025122 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:21.089553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545c992-8751-4c10-b8b3-53344ea9e3c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:21.090118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545c992-8751-4c10-b8b3-53344ea9e3c1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:21.090398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545c992-8751-4c10-b8b3-53344ea9e3c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:21.091792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545c992-8751-4c10-b8b3-53344ea9e3c1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:21.095521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:21.103085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:21.103955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:21.104431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:21.104834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:21.105664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] RBAC: Adding query filter params (name=member) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:21.106025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f545c992-8751-4c10-b8b3-53344ea9e3c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:21.106253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:21.107921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-f545c992-8751-4c10-b8b3-53344ea9e3c1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:21.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:21.112901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:21.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 157/304] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:21.122376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545c992-8751-4c10-b8b3-53344ea9e3c1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:21.126274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 148/305] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:21.131725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:21.249128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545c992-8751-4c10-b8b3-53344ea9e3c1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:21.264471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:21.278451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff45faa-04f4-422b-a98d-57a6c3af9bfa None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:21.294544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:21.295571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:21.296167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:21.296698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:21.297610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] RBAC: Adding query filter params (name=member) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:21.298193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:21.299969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:21.309961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 158/306] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:27:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:21.315389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:21.320134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6e8109-e749-4bcb-b156-dced0b9239bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:21.320507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6e8109-e749-4bcb-b156-dced0b9239bc None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:21.320790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6e8109-e749-4bcb-b156-dced0b9239bc None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:21.321061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6e8109-e749-4bcb-b156-dced0b9239bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:21.396476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:21.411664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6e8109-e749-4bcb-b156-dced0b9239bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:21.413499 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6e8109-e749-4bcb-b156-dced0b9239bc None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:21.455333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4989a7d-6739-4355-91fb-3890d774cc3d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:21.503045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4961ae80-215e-43a8-b536-a0544af2136e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:21.503706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4961ae80-215e-43a8-b536-a0544af2136e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:21.503992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4961ae80-215e-43a8-b536-a0544af2136e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:21.504272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4961ae80-215e-43a8-b536-a0544af2136e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:21.520112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4961ae80-215e-43a8-b536-a0544af2136e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:21.524428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-4961ae80-215e-43a8-b536-a0544af2136e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:21.528682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:21.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:21.531692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:21.534981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 159/307] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:27:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:21.536265 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4961ae80-215e-43a8-b536-a0544af2136e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:21.542733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 149/308] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:21.543733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6e8109-e749-4bcb-b156-dced0b9239bc None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:21.670397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6e8109-e749-4bcb-b156-dced0b9239bc None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:21.685881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4961ae80-215e-43a8-b536-a0544af2136e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:21.710933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:21.713820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:21.714775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:21.716076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:21.716560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:21.717440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:21.717970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:21.719755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:21.734211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 150/309] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:21.736908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:21.742328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17676cb-5b85-41b3-ba5b-a4f8156a1793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:21.742994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17676cb-5b85-41b3-ba5b-a4f8156a1793 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:21.743271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17676cb-5b85-41b3-ba5b-a4f8156a1793 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:21.743539 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17676cb-5b85-41b3-ba5b-a4f8156a1793 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:21.815514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:21.826049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17676cb-5b85-41b3-ba5b-a4f8156a1793 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:21.828091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17676cb-5b85-41b3-ba5b-a4f8156a1793 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:21.887527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86af388-cad2-4fef-a06d-c10b4896c4f9 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:21.930100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89559fb1-20da-4fa0-8b96-28f1414b80d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:21.931169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89559fb1-20da-4fa0-8b96-28f1414b80d6 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:21.931658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89559fb1-20da-4fa0-8b96-28f1414b80d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:21.932121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89559fb1-20da-4fa0-8b96-28f1414b80d6 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:21.933603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89559fb1-20da-4fa0-8b96-28f1414b80d6 None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:21.935323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89559fb1-20da-4fa0-8b96-28f1414b80d6 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:21.938333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 151/310] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:21 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:21.939406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89559fb1-20da-4fa0-8b96-28f1414b80d6 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:21.944259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 160/311] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:21.947558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17676cb-5b85-41b3-ba5b-a4f8156a1793 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:21.969573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:22.058469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17676cb-5b85-41b3-ba5b-a4f8156a1793 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:22.068084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89559fb1-20da-4fa0-8b96-28f1414b80d6 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:22.096300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:22.097322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f052b2b5-2cc9-4c9f-9950-51fb707e0032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:22.101275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f052b2b5-2cc9-4c9f-9950-51fb707e0032 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.101275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f052b2b5-2cc9-4c9f-9950-51fb707e0032 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.101275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f052b2b5-2cc9-4c9f-9950-51fb707e0032 None admin] PATH_INFO: `/v3/groups/90d2140ee6e84bbfb742def6466833e8` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.101275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f052b2b5-2cc9-4c9f-9950-51fb707e0032 None admin] RBAC: Authorizing `identity:get_group(group_id=90d2140ee6e84bbfb742def6466833e8)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:22.101588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f052b2b5-2cc9-4c9f-9950-51fb707e0032 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:22.103679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 161/312] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Oct 19 00:27:21 2022] GET /identity/v3/groups/90d2140ee6e84bbfb742def6466833e8 => generated 226 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:22.104460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f052b2b5-2cc9-4c9f-9950-51fb707e0032 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:22.124517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d425feb6-e9dd-49d7-ab93-67e98ebb6728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:22.125666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425feb6-e9dd-49d7-ab93-67e98ebb6728 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.126252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425feb6-e9dd-49d7-ab93-67e98ebb6728 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.126791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425feb6-e9dd-49d7-ab93-67e98ebb6728 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.203401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:22.221263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d425feb6-e9dd-49d7-ab93-67e98ebb6728 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:22.221474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f052b2b5-2cc9-4c9f-9950-51fb707e0032 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:22.223608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d425feb6-e9dd-49d7-ab93-67e98ebb6728 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:22.261338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280aa4f-3340-4dde-910d-e230335d262f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:22.262013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280aa4f-3340-4dde-910d-e230335d262f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.262245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280aa4f-3340-4dde-910d-e230335d262f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.262538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280aa4f-3340-4dde-910d-e230335d262f None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.263844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280aa4f-3340-4dde-910d-e230335d262f None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:22.266239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280aa4f-3340-4dde-910d-e230335d262f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:22.267984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 162/313] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:22 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:22.268764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280aa4f-3340-4dde-910d-e230335d262f None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:22.360827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 152/314] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:22.362463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d425feb6-e9dd-49d7-ab93-67e98ebb6728 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:22.386970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:22.399009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280aa4f-3340-4dde-910d-e230335d262f None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:22.444478 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba9785e-cafa-44c1-8166-264cfc73ae16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:22.445096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba9785e-cafa-44c1-8166-264cfc73ae16 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.445375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba9785e-cafa-44c1-8166-264cfc73ae16 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.445645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba9785e-cafa-44c1-8166-264cfc73ae16 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.446383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba9785e-cafa-44c1-8166-264cfc73ae16 None admin] RBAC: Adding query filter params (group.id=90d2140ee6e84bbfb742def6466833e8, role.id=d729ee72a6d746959af49c5cab72ddd6, scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:22.446716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba9785e-cafa-44c1-8166-264cfc73ae16 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:22.448168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba9785e-cafa-44c1-8166-264cfc73ae16 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:22.458271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 163/315] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Oct 19 00:27:22 2022] GET /identity/v3/role_assignments?group.id=90d2140ee6e84bbfb742def6466833e8&scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362&role.id=d729ee72a6d746959af49c5cab72ddd6 => generated 611 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:22.460098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1942b2c6-869d-4963-adff-8d5a238a4b83 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.460396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1942b2c6-869d-4963-adff-8d5a238a4b83 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.460673 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1942b2c6-869d-4963-adff-8d5a238a4b83 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.462128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 164/316] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:22.462753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1942b2c6-869d-4963-adff-8d5a238a4b83 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:22.488370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d425feb6-e9dd-49d7-ab93-67e98ebb6728 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:22.508151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:22.531582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4870e2-5351-47a4-94d3-8944bb724880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:22.532544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4870e2-5351-47a4-94d3-8944bb724880 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.533043 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4870e2-5351-47a4-94d3-8944bb724880 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.533471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4870e2-5351-47a4-94d3-8944bb724880 None admin] PATH_INFO: `/v3/groups/90d2140ee6e84bbfb742def6466833e8` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.535043 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4870e2-5351-47a4-94d3-8944bb724880 None admin] RBAC: Authorizing `identity:get_group(group_id=90d2140ee6e84bbfb742def6466833e8)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:22.536780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4870e2-5351-47a4-94d3-8944bb724880 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:22.539029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 153/317] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Oct 19 00:27:22 2022] GET /identity/v3/groups/90d2140ee6e84bbfb742def6466833e8 => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:22.540548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1f396-8dad-477c-9722-3fb1b06ede6a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.541025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1f396-8dad-477c-9722-3fb1b06ede6a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.541530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1f396-8dad-477c-9722-3fb1b06ede6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.564411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-56c1f396-8dad-477c-9722-3fb1b06ede6a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:22.565694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-56c1f396-8dad-477c-9722-3fb1b06ede6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:22.618071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1942b2c6-869d-4963-adff-8d5a238a4b83 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:22.653911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 154/318] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:22.659596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c1f396-8dad-477c-9722-3fb1b06ede6a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:22.665868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9345a167-fb5e-429c-bee3-b1771739ba97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:22.666879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9345a167-fb5e-429c-bee3-b1771739ba97 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.667373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9345a167-fb5e-429c-bee3-b1771739ba97 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.668846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9345a167-fb5e-429c-bee3-b1771739ba97 None admin] PATH_INFO: `/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.670295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9345a167-fb5e-429c-bee3-b1771739ba97 None admin] RBAC: Authorizing `identity:get_project(project_id=c2fc9c9b4fba4ad6954f2d7aaba3b362)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:22.672565 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9345a167-fb5e-429c-bee3-b1771739ba97 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:22.675264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 165/319] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:22 2022] GET /identity/v3/projects/c2fc9c9b4fba4ad6954f2d7aaba3b362 => generated 298 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:22.681503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9345a167-fb5e-429c-bee3-b1771739ba97 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:22.697001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:22.795718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:22.822580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9345a167-fb5e-429c-bee3-b1771739ba97 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:22.826222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c1f396-8dad-477c-9722-3fb1b06ede6a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:22.864190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e010baf-c764-4ee3-82fc-ee27786f3193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:22.864842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e010baf-c764-4ee3-82fc-ee27786f3193 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.865444 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e010baf-c764-4ee3-82fc-ee27786f3193 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.865786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e010baf-c764-4ee3-82fc-ee27786f3193 None admin] PATH_INFO: `/v3/groups/90d2140ee6e84bbfb742def6466833e8` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.867356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e010baf-c764-4ee3-82fc-ee27786f3193 None admin] RBAC: Authorizing `identity:get_group(group_id=90d2140ee6e84bbfb742def6466833e8)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:22.869486 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e010baf-c764-4ee3-82fc-ee27786f3193 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:22.872187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 166/320] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Oct 19 00:27:22 2022] GET /identity/v3/groups/90d2140ee6e84bbfb742def6466833e8 => generated 226 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:22.879007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e010baf-c764-4ee3-82fc-ee27786f3193 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:22.887567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1a5e03-9d02-40d8-82bc-212405d1f140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:22.888195 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a5e03-9d02-40d8-82bc-212405d1f140 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:22.888535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a5e03-9d02-40d8-82bc-212405d1f140 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:22.888811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a5e03-9d02-40d8-82bc-212405d1f140 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:22.889557 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1a5e03-9d02-40d8-82bc-212405d1f140 None admin] RBAC: Adding query filter params (group.id=90d2140ee6e84bbfb742def6466833e8, role.id=d729ee72a6d746959af49c5cab72ddd6, scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:22.890689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1a5e03-9d02-40d8-82bc-212405d1f140 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:22.892183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1a5e03-9d02-40d8-82bc-212405d1f140 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:22.899678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 155/321] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Oct 19 00:27:22 2022] GET /identity/v3/role_assignments?group.id=90d2140ee6e84bbfb742def6466833e8&scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca&role.id=d729ee72a6d746959af49c5cab72ddd6 => generated 611 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:23.006379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e010baf-c764-4ee3-82fc-ee27786f3193 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:23.028712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586bda0c-523e-4958-9dbd-fde33f34c1c3 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:23.029058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586bda0c-523e-4958-9dbd-fde33f34c1c3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:23.029255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586bda0c-523e-4958-9dbd-fde33f34c1c3 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:23.030776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 156/322] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:27:23.037204 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad26c42b-efec-4f46-959c-1981343efc87 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:23.037571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad26c42b-efec-4f46-959c-1981343efc87 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:23.037732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad26c42b-efec-4f46-959c-1981343efc87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:23.058390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:23.058975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:23.059252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:23.059536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:23.060273 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] RBAC: Adding query filter params (group.id=90d2140ee6e84bbfb742def6466833e8, role.id=66029780d52042b9a44109eb91c0d80e, scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:23.060571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:23.062638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:23.067678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ad26c42b-efec-4f46-959c-1981343efc87 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:23.068964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ad26c42b-efec-4f46-959c-1981343efc87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:23.069250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 167/323] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Oct 19 00:27:22 2022] GET /identity/v3/role_assignments?group.id=90d2140ee6e84bbfb742def6466833e8&scope.project.id=c2fc9c9b4fba4ad6954f2d7aaba3b362&role.id=66029780d52042b9a44109eb91c0d80e => generated 611 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:23.070089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:23.097643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:23.170136 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 157/324] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:23.202044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb26a6-a2a6-4342-80d3-0e1b278d0ab7 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:23.247157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9387983a-e53d-4928-bca0-5ec8f1a611cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:23.247825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9387983a-e53d-4928-bca0-5ec8f1a611cc None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:23.248160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9387983a-e53d-4928-bca0-5ec8f1a611cc None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:23.248439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9387983a-e53d-4928-bca0-5ec8f1a611cc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:23.261952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9387983a-e53d-4928-bca0-5ec8f1a611cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-9387983a-e53d-4928-bca0-5ec8f1a611cc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:23.267853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:23.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:23.269909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 168/325] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:27:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:23.276661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad26c42b-efec-4f46-959c-1981343efc87 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:23.295239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:23.394631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad26c42b-efec-4f46-959c-1981343efc87 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:23.432456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:23.433084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:23.433394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:23.433696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:23.434383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:23.435181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:23.436138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:23.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 158/326] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:23 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:23.450509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9387983a-e53d-4928-bca0-5ec8f1a611cc None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:23.475502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:23.548220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:23.572082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:23.581313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9387983a-e53d-4928-bca0-5ec8f1a611cc None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:23.608852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f2e87-ebcc-42c8-a6b6-94a93e5d5b7f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:23.625341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a05be3-82a5-49ae-8c55-38d5bc417157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:23.626177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a05be3-82a5-49ae-8c55-38d5bc417157 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:23.626608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a05be3-82a5-49ae-8c55-38d5bc417157 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:23.626880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a05be3-82a5-49ae-8c55-38d5bc417157 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:23.658053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f66fd-58cf-4c3e-b6a3-ef8ccf5faa92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:23.658784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f66fd-58cf-4c3e-b6a3-ef8ccf5faa92 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:23.659141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f66fd-58cf-4c3e-b6a3-ef8ccf5faa92 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:23.659555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f66fd-58cf-4c3e-b6a3-ef8ccf5faa92 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:23.673668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65f66fd-58cf-4c3e-b6a3-ef8ccf5faa92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-b65f66fd-58cf-4c3e-b6a3-ef8ccf5faa92 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:23.679377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:23.680306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:23.682105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 159/327] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:27:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:23.686768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f66fd-58cf-4c3e-b6a3-ef8ccf5faa92 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:23.720902 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a05be3-82a5-49ae-8c55-38d5bc417157 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:23.722677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a05be3-82a5-49ae-8c55-38d5bc417157 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:23.821098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:23.822422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65f66fd-58cf-4c3e-b6a3-ef8ccf5faa92 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:23.850790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 169/328] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:23.856121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a05be3-82a5-49ae-8c55-38d5bc417157 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:23.861154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:23.861770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:23.862046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:23.862324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:23.863687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:23.863687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:23.865731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:23.866764 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:23.878590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 160/329] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:23.883409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:23.972412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a05be3-82a5-49ae-8c55-38d5bc417157 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:23.998952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60de3ec6-bd81-4a1f-9bb2-6898ab0a6db4 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:24.021852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b3c888-7f6e-4500-bc8d-ee154f068c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:24.022305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3c888-7f6e-4500-bc8d-ee154f068c09 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:24.022665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3c888-7f6e-4500-bc8d-ee154f068c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:24.022944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3c888-7f6e-4500-bc8d-ee154f068c09 None admin] PATH_INFO: `/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:24.024081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b3c888-7f6e-4500-bc8d-ee154f068c09 None admin] RBAC: Authorizing `identity:get_project(project_id=fd8ddc1c4f0c4476944b52ad5fa0d6ca)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:24.025525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b3c888-7f6e-4500-bc8d-ee154f068c09 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:24.027631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 170/330] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:23 2022] GET /identity/v3/projects/fd8ddc1c4f0c4476944b52ad5fa0d6ca => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:24.032915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b3c888-7f6e-4500-bc8d-ee154f068c09 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:24.043969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f20917-e3bd-43cd-ad05-53485e606420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:24.044843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f20917-e3bd-43cd-ad05-53485e606420 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:24.045261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f20917-e3bd-43cd-ad05-53485e606420 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:24.045681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f20917-e3bd-43cd-ad05-53485e606420 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:24.088991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:24.117514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:24.124893 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f20917-e3bd-43cd-ad05-53485e606420 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:24.126373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f20917-e3bd-43cd-ad05-53485e606420 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:24.144480 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b3c888-7f6e-4500-bc8d-ee154f068c09 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:24.181442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00156f67-1eda-4c86-abe0-30f7c6e1fa56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:24.182101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00156f67-1eda-4c86-abe0-30f7c6e1fa56 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:24.182384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00156f67-1eda-4c86-abe0-30f7c6e1fa56 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:24.182708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00156f67-1eda-4c86-abe0-30f7c6e1fa56 None admin] PATH_INFO: `/v3/groups/90d2140ee6e84bbfb742def6466833e8` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:24.183906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00156f67-1eda-4c86-abe0-30f7c6e1fa56 None admin] RBAC: Authorizing `identity:get_group(group_id=90d2140ee6e84bbfb742def6466833e8)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:24.185384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00156f67-1eda-4c86-abe0-30f7c6e1fa56 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:24.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 171/331] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Oct 19 00:27:24 2022] GET /identity/v3/groups/90d2140ee6e84bbfb742def6466833e8 => generated 226 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:24.193266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00156f67-1eda-4c86-abe0-30f7c6e1fa56 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:24.234228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 161/332] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 19 00:27:24.240025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f20917-e3bd-43cd-ad05-53485e606420 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:24.301688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00156f67-1eda-4c86-abe0-30f7c6e1fa56 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:24.316712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:24.338801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:24.339340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:24.339469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:24.339674 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:24.339951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:24.340766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] RBAC: Adding query filter params (group.id=90d2140ee6e84bbfb742def6466833e8, role.id=66029780d52042b9a44109eb91c0d80e, scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:24.341113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:24.342786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:24.347267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f20917-e3bd-43cd-ad05-53485e606420 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:24.350060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 172/333] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Oct 19 00:27:24 2022] GET /identity/v3/role_assignments?group.id=90d2140ee6e84bbfb742def6466833e8&scope.project.id=fd8ddc1c4f0c4476944b52ad5fa0d6ca&role.id=66029780d52042b9a44109eb91c0d80e => generated 611 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:24.387914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9984fbdb-dc5e-4314-a5ae-c21f7d6afb83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:24.388573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9984fbdb-dc5e-4314-a5ae-c21f7d6afb83 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:24.388840 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9984fbdb-dc5e-4314-a5ae-c21f7d6afb83 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:24.389112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9984fbdb-dc5e-4314-a5ae-c21f7d6afb83 None admin] PATH_INFO: `/v3/projects/d417ad1948a04e309383739147354388` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:24.391157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9984fbdb-dc5e-4314-a5ae-c21f7d6afb83 None admin] RBAC: Authorizing `identity:get_project(project_id=d417ad1948a04e309383739147354388)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:24.392649 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9984fbdb-dc5e-4314-a5ae-c21f7d6afb83 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:24.394351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 162/334] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:27:24 2022] GET /identity/v3/projects/d417ad1948a04e309383739147354388 => generated 344 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:24.400027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:24.511774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed74260-150f-4172-adf4-4193c5948702 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:24.551789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dc435e-579e-4a14-a08f-f334acfbe51b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:24.554522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc435e-579e-4a14-a08f-f334acfbe51b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:24.554522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc435e-579e-4a14-a08f-f334acfbe51b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:24.554522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc435e-579e-4a14-a08f-f334acfbe51b None admin] PATH_INFO: `/v3/groups/052736fb793e40ab86a6719651d30372` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:24.554522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dc435e-579e-4a14-a08f-f334acfbe51b None admin] RBAC: Authorizing `identity:get_group(group_id=052736fb793e40ab86a6719651d30372)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:24.555534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dc435e-579e-4a14-a08f-f334acfbe51b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:24.557110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 173/335] 10.222.0.26 () {56 vars in 1180 bytes} [Wed Oct 19 00:27:24 2022] GET /identity/v3/groups/052736fb793e40ab86a6719651d30372 => generated 229 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:24.563563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9984fbdb-dc5e-4314-a5ae-c21f7d6afb83 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:24.668974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9984fbdb-dc5e-4314-a5ae-c21f7d6afb83 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:24.706475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23037f2c-08a1-469f-af2a-d2a86eaeaf21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:24.707109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23037f2c-08a1-469f-af2a-d2a86eaeaf21 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:24.707395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23037f2c-08a1-469f-af2a-d2a86eaeaf21 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:24.707666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23037f2c-08a1-469f-af2a-d2a86eaeaf21 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:24.708396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23037f2c-08a1-469f-af2a-d2a86eaeaf21 None admin] RBAC: Adding query filter params (group.id=052736fb793e40ab86a6719651d30372, role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=d417ad1948a04e309383739147354388) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:24.708704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23037f2c-08a1-469f-af2a-d2a86eaeaf21 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:24.710190 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23037f2c-08a1-469f-af2a-d2a86eaeaf21 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:24.716424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 163/336] 10.222.0.26 () {56 vars in 1376 bytes} [Wed Oct 19 00:27:24 2022] GET /identity/v3/role_assignments?group.id=052736fb793e40ab86a6719651d30372&scope.project.id=d417ad1948a04e309383739147354388&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 611 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:27.587236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d292d2-d2a1-4b4d-af72-2fe18f677445 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:27.588234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d292d2-d2a1-4b4d-af72-2fe18f677445 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:27.588720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d292d2-d2a1-4b4d-af72-2fe18f677445 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:27.590613 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 174/337] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:27.596931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f20e9d4-b426-4154-b982-904658bee935 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:27.597558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f20e9d4-b426-4154-b982-904658bee935 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:27.597558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f20e9d4-b426-4154-b982-904658bee935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:27.627713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-3f20e9d4-b426-4154-b982-904658bee935 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:27.627938 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1d9f5-a175-4c81-beed-bf29b5b1963b None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:27.628438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1d9f5-a175-4c81-beed-bf29b5b1963b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:27.628937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1d9f5-a175-4c81-beed-bf29b5b1963b None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:27.629040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3f20e9d4-b426-4154-b982-904658bee935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:27.631026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 175/338] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:27.639637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154dab9a-e6ab-4515-8168-d695ca4395ec None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:27.640794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154dab9a-e6ab-4515-8168-d695ca4395ec None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:27.641264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154dab9a-e6ab-4515-8168-d695ca4395ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:27.666145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:27.673454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-154dab9a-e6ab-4515-8168-d695ca4395ec None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:27.674833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-154dab9a-e6ab-4515-8168-d695ca4395ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:27.742783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:27.752693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 164/339] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:27.754582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1619ae-6348-4d3c-8793-0b3f578eef48 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:27.755040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1619ae-6348-4d3c-8793-0b3f578eef48 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:27.755453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1619ae-6348-4d3c-8793-0b3f578eef48 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:27.756883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 165/340] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:27.763929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff08aae-56cf-4d0c-bccb-d1b90ac37c83 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:27.764155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff08aae-56cf-4d0c-bccb-d1b90ac37c83 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:27.764781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff08aae-56cf-4d0c-bccb-d1b90ac37c83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:27.787233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8ff08aae-56cf-4d0c-bccb-d1b90ac37c83 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:27.788757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8ff08aae-56cf-4d0c-bccb-d1b90ac37c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:27.809008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 176/341] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:27.893140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 166/342] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:28.127649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154dab9a-e6ab-4515-8168-d695ca4395ec None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:28.143989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff08aae-56cf-4d0c-bccb-d1b90ac37c83 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:28.273597 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154dab9a-e6ab-4515-8168-d695ca4395ec None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:28.279333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:28.295354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff08aae-56cf-4d0c-bccb-d1b90ac37c83 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:28.315290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a98b7-1f58-4ab6-b6ca-3df96c1524c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:28.316225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a98b7-1f58-4ab6-b6ca-3df96c1524c9 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.316511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a98b7-1f58-4ab6-b6ca-3df96c1524c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.316787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a98b7-1f58-4ab6-b6ca-3df96c1524c9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.328778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a98b7-1f58-4ab6-b6ca-3df96c1524c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:28.330217 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a98b7-1f58-4ab6-b6ca-3df96c1524c9 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:28.333088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:28.333720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.333992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.334258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.344714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:28.346153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-ac8a98b7-1f58-4ab6-b6ca-3df96c1524c9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:28.348677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:28.349591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:28.350435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:28.350435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:28.350435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:28.350435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 177/343] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:28.356283 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75ab791-2559-440e-9b18-5a0c9a7dceef None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.356569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75ab791-2559-440e-9b18-5a0c9a7dceef None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.356857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75ab791-2559-440e-9b18-5a0c9a7dceef None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.363139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 178/344] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:28 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:28.363139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75ab791-2559-440e-9b18-5a0c9a7dceef None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:28.364319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:28.365194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:28.366497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:28.366497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:28.366497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:28.366497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 167/345] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:28.366910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:28.397832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:28.480822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37a881f-1183-4b87-bf71-8a9f991acc97 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:28.482968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75ab791-2559-440e-9b18-5a0c9a7dceef None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:28.518864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fba2b3-ce85-4586-9e14-27041bfd1e2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:28.519483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fba2b3-ce85-4586-9e14-27041bfd1e2a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.519761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fba2b3-ce85-4586-9e14-27041bfd1e2a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.520067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fba2b3-ce85-4586-9e14-27041bfd1e2a None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.520674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fba2b3-ce85-4586-9e14-27041bfd1e2a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:28.520967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fba2b3-ce85-4586-9e14-27041bfd1e2a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:28.521187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:28.521823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.522102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.522349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fba2b3-ce85-4586-9e14-27041bfd1e2a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:28.522442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.533815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:28.534756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:28.535302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:28.544158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 168/346] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:28.547373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e976cf22-ce2d-4f96-9002-8fe8a0812564 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.548204 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e976cf22-ce2d-4f96-9002-8fe8a0812564 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.548662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e976cf22-ce2d-4f96-9002-8fe8a0812564 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:28.553264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:28.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:28.560090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:28.560090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:28.560090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:28.560090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 179/347] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:28.560090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:28.573580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e976cf22-ce2d-4f96-9002-8fe8a0812564 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:28.575067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e976cf22-ce2d-4f96-9002-8fe8a0812564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:28.647631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:28.666610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 169/348] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:28.666610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e976cf22-ce2d-4f96-9002-8fe8a0812564 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:28.684406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d86bd0a-b144-4592-9401-92183c817f00 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:28.731470 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:28.732661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.733121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.734198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.735653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:28.736363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:28.738722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:28.760681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 180/349] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:28.760681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:28.785870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e976cf22-ce2d-4f96-9002-8fe8a0812564 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:28.818353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:28.824892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf517e67-dea1-44e8-8762-c7c69ebc13c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:28.825718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf517e67-dea1-44e8-8762-c7c69ebc13c5 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.826222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf517e67-dea1-44e8-8762-c7c69ebc13c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.826751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf517e67-dea1-44e8-8762-c7c69ebc13c5 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.828452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf517e67-dea1-44e8-8762-c7c69ebc13c5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:28.830288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf517e67-dea1-44e8-8762-c7c69ebc13c5 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:28.863053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 170/350] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:27:28 2022] POST /identity/v3/users => generated 246 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:28.864490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf517e67-dea1-44e8-8762-c7c69ebc13c5 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:28.884879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8af8d-f97f-4665-b484-152b8289df88 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:28.929482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:28.930000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:28.930281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:28.930579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:28.931270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:28.931579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:28.933048 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:28.958086 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:28.964315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 181/351] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:28.970313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:29.054411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf517e67-dea1-44e8-8762-c7c69ebc13c5 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:29.102364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598a2ac5-2f8c-4ab9-8982-da518bcf6ae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:29.103023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a2ac5-2f8c-4ab9-8982-da518bcf6ae0 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:29.103308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a2ac5-2f8c-4ab9-8982-da518bcf6ae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:29.103578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a2ac5-2f8c-4ab9-8982-da518bcf6ae0 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:29.104234 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598a2ac5-2f8c-4ab9-8982-da518bcf6ae0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:29.105679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598a2ac5-2f8c-4ab9-8982-da518bcf6ae0 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:29.112099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84961213-3317-42a6-be1c-5303ed32a480 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:29.133988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 171/352] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:27:28 2022] POST /identity/v3/users => generated 248 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:29.135500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598a2ac5-2f8c-4ab9-8982-da518bcf6ae0 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:29.162238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf83e1-33aa-4a98-bf63-b4eb88beec02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:29.163305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf83e1-33aa-4a98-bf63-b4eb88beec02 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:29.163804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf83e1-33aa-4a98-bf63-b4eb88beec02 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:29.164286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf83e1-33aa-4a98-bf63-b4eb88beec02 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:29.181586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cf83e1-33aa-4a98-bf63-b4eb88beec02 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:29.183706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cf83e1-33aa-4a98-bf63-b4eb88beec02 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-d3cf83e1-33aa-4a98-bf63-b4eb88beec02 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:29.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:29.205150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:29.206772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:29.206772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:29.206772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:29.206772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 182/353] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:29.212210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c034e9-a316-49bd-8688-ba7799651f7a None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:29.212690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c034e9-a316-49bd-8688-ba7799651f7a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:29.212860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c034e9-a316-49bd-8688-ba7799651f7a None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:29.214394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 183/354] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:29 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:29.215437 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c034e9-a316-49bd-8688-ba7799651f7a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:29.227002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:29.301621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:29.346019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598a2ac5-2f8c-4ab9-8982-da518bcf6ae0 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:29.369978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c034e9-a316-49bd-8688-ba7799651f7a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:29.395691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce8ad3e-0401-46d1-a362-07e742abaa02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:29.396314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce8ad3e-0401-46d1-a362-07e742abaa02 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:29.396595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce8ad3e-0401-46d1-a362-07e742abaa02 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:29.396865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce8ad3e-0401-46d1-a362-07e742abaa02 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:29.398003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce8ad3e-0401-46d1-a362-07e742abaa02 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:29.399934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce8ad3e-0401-46d1-a362-07e742abaa02 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:29.414164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:29.415003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:29.415274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:29.415549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:29.416369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:29.416671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:29.418101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:29.426406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 172/355] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:27:29 2022] POST /identity/v3/users => generated 248 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:29.428019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975496d6-d155-41c5-8224-e37dce640584 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:29.428370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975496d6-d155-41c5-8224-e37dce640584 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:29.428796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975496d6-d155-41c5-8224-e37dce640584 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:29.432832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 184/356] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:29.443661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:29.451160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-975496d6-d155-41c5-8224-e37dce640584 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:29.453402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-975496d6-d155-41c5-8224-e37dce640584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:29.560385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 173/357] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:29.612743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98db7d0d-5b13-4aa8-a8c3-40d604464d7f None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:29.628180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:29.654079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f0015-0f55-4e9e-aa9e-1bc51d945333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:29.654940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f0015-0f55-4e9e-aa9e-1bc51d945333 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:29.655160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f0015-0f55-4e9e-aa9e-1bc51d945333 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:29.655454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f0015-0f55-4e9e-aa9e-1bc51d945333 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:29.656206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313f0015-0f55-4e9e-aa9e-1bc51d945333 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:29.657672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313f0015-0f55-4e9e-aa9e-1bc51d945333 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:29.683598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 185/358] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:27:29 2022] POST /identity/v3/users => generated 249 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:29.878438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975496d6-d155-41c5-8224-e37dce640584 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:29.911587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:30.007957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975496d6-d155-41c5-8224-e37dce640584 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:30.046590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:30.047310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:30.047590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:30.047856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:30.048554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:30.050024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:30.059323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.sql.core [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 00:27:30.059323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 19 00:27:30.059323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [parameters: {'id': '354b3831baef43768f12a88686c2bbe5', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 19 00:27:30.059323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81812) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:27:30.067336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return ref.to_dict() Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:27:30.068221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application yield resource Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise self.value Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application session.commit() Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 00:27:30.069113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.flush() Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application compat.raise_( Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 00:27:30.069947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:27:30.071946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:27:30.073836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [parameters: {'id': '354b3831baef43768f12a88686c2bbe5', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:30.075775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 19 00:27:30.077986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:30.080164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 174/359] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Oct 19 00:27:29 2022] POST /identity/v3/roles => generated 144 bytes in 191 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:30.080308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f0015-0f55-4e9e-aa9e-1bc51d945333 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:30.237564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f0015-0f55-4e9e-aa9e-1bc51d945333 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:30.278636 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f30bdb3-6fe7-4046-bf5c-290481b0ad69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:30.279279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f30bdb3-6fe7-4046-bf5c-290481b0ad69 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:30.279561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f30bdb3-6fe7-4046-bf5c-290481b0ad69 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:30.279838 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f30bdb3-6fe7-4046-bf5c-290481b0ad69 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:30.292035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f30bdb3-6fe7-4046-bf5c-290481b0ad69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-9f30bdb3-6fe7-4046-bf5c-290481b0ad69 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:30.298854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:30.300996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:30.303118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 186/360] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:27:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:30.306514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:30.448080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:30.463173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba2c229-9a0f-468b-a98a-aa8a0382c761 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:30.504925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53003ae-f650-4a25-853b-1d83f3efec67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:30.505563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53003ae-f650-4a25-853b-1d83f3efec67 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:30.505846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53003ae-f650-4a25-853b-1d83f3efec67 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:30.506124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53003ae-f650-4a25-853b-1d83f3efec67 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:30.506863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53003ae-f650-4a25-853b-1d83f3efec67 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:30.507171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53003ae-f650-4a25-853b-1d83f3efec67 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:30.508634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53003ae-f650-4a25-853b-1d83f3efec67 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:30.519155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 175/361] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:31.830588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d520480-a761-4fd8-84f1-2c7119173b23 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:31.831559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d520480-a761-4fd8-84f1-2c7119173b23 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:31.831559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d520480-a761-4fd8-84f1-2c7119173b23 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:31.832659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 187/362] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:31.838521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164aa46f-d74c-4adb-9c7d-5814f8f6cb32 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:31.839421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164aa46f-d74c-4adb-9c7d-5814f8f6cb32 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:31.839421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164aa46f-d74c-4adb-9c7d-5814f8f6cb32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:31.866822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-164aa46f-d74c-4adb-9c7d-5814f8f6cb32 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:31.868066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-164aa46f-d74c-4adb-9c7d-5814f8f6cb32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:31.944580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 176/363] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:32.221721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d520480-a761-4fd8-84f1-2c7119173b23 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:32.263132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:32.379134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d520480-a761-4fd8-84f1-2c7119173b23 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:32.425883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a50f46-55b0-4a2f-bf72-b64a225c0a4d None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:32.426882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a50f46-55b0-4a2f-bf72-b64a225c0a4d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:32.427688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a50f46-55b0-4a2f-bf72-b64a225c0a4d None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:32.431027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 177/364] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:32 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:32.439587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41b4dc8-e73a-493d-b5f0-3562d49d1ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:32.440252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b4dc8-e73a-493d-b5f0-3562d49d1ded None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:32.440537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b4dc8-e73a-493d-b5f0-3562d49d1ded None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:32.440811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b4dc8-e73a-493d-b5f0-3562d49d1ded None admin] PATH_INFO: `/v3/roles/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:32.442747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713ee24-ef24-470f-9c13-1d03d8aa4f8d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:32.443504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713ee24-ef24-470f-9c13-1d03d8aa4f8d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:32.444225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713ee24-ef24-470f-9c13-1d03d8aa4f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:32.458971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41b4dc8-e73a-493d-b5f0-3562d49d1ded None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-f41b4dc8-e73a-493d-b5f0-3562d49d1ded None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:32.463955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:32.465631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:32.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 188/365] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:32.471623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41b4dc8-e73a-493d-b5f0-3562d49d1ded None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:32.480805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d713ee24-ef24-470f-9c13-1d03d8aa4f8d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:32.483821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d713ee24-ef24-470f-9c13-1d03d8aa4f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:32.613238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:32.627504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41b4dc8-e73a-493d-b5f0-3562d49d1ded None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:32.645364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 178/366] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:32.646672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e98c079-5bea-4023-8543-03fb6ea72cae None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:32.646955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e98c079-5bea-4023-8543-03fb6ea72cae None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:32.647249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e98c079-5bea-4023-8543-03fb6ea72cae None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:32.648822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 179/367] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:32.662515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4098858f-3ed4-4ef2-a2b7-8b98eb618a08 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:32.662515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4098858f-3ed4-4ef2-a2b7-8b98eb618a08 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:32.662515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4098858f-3ed4-4ef2-a2b7-8b98eb618a08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:32.691909 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-4098858f-3ed4-4ef2-a2b7-8b98eb618a08 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:32.693915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4098858f-3ed4-4ef2-a2b7-8b98eb618a08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:32.713991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:32.731947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:32.733934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:32.734717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:32.735454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:32.736913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] RBAC: Adding query filter params (name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:32.737708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:32.741263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:32.756385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 189/368] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:32.762020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:32.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 180/369] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:32.916920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d108ad9a-ab9a-40ae-86a7-ef46cb74688f None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:32.972444 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a1b8a4-60e9-46ed-8d73-a6b402246449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:32.973730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1b8a4-60e9-46ed-8d73-a6b402246449 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:32.974199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1b8a4-60e9-46ed-8d73-a6b402246449 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:32.974682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a1b8a4-60e9-46ed-8d73-a6b402246449 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:32.994386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a1b8a4-60e9-46ed-8d73-a6b402246449 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:32.996208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a1b8a4-60e9-46ed-8d73-a6b402246449 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:33.013901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a185885d-edd8-45ce-95de-96aeeb7553a5 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a185885d-edd8-45ce-95de-96aeeb7553a5 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.014644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a185885d-edd8-45ce-95de-96aeeb7553a5 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.016405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 181/370] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-31a1b8a4-60e9-46ed-8d73-a6b402246449 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:33.019360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:33.020269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:33.021114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:33.021114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:33.021114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:33.021114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 190/371] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:33.027863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a185885d-edd8-45ce-95de-96aeeb7553a5 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:33.028623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c62cb2-891b-4ee6-80ae-f7c3d753af1e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.029078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c62cb2-891b-4ee6-80ae-f7c3d753af1e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.029513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c62cb2-891b-4ee6-80ae-f7c3d753af1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.047208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:33.063562 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-10c62cb2-891b-4ee6-80ae-f7c3d753af1e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:33.066795 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-10c62cb2-891b-4ee6-80ae-f7c3d753af1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:33.176772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 191/372] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:33.231471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c62cb2-891b-4ee6-80ae-f7c3d753af1e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:33.250451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a185885d-edd8-45ce-95de-96aeeb7553a5 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:33.316380 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:33.317427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.317937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.318425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.319665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:33.320209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:33.322829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:33.346340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 182/373] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:33.348321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:33.356775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:33.366271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c62cb2-891b-4ee6-80ae-f7c3d753af1e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:33.398329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:33.411104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e93a50-ec51-42ae-910d-fb52f679396f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:33.412514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e93a50-ec51-42ae-910d-fb52f679396f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.413005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e93a50-ec51-42ae-910d-fb52f679396f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.413502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e93a50-ec51-42ae-910d-fb52f679396f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.441810 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e93a50-ec51-42ae-910d-fb52f679396f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-84e93a50-ec51-42ae-910d-fb52f679396f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:33.449338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:33.450305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:33.451151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 192/374] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:33.452919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e93a50-ec51-42ae-910d-fb52f679396f None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:33.539570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5af22-1a6b-4883-a485-a007d0cc24c3 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:33.566715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e93a50-ec51-42ae-910d-fb52f679396f None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:33.610597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bb6b9-c0f9-4a10-b29c-ce66e96559cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:33.611725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bb6b9-c0f9-4a10-b29c-ce66e96559cc None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.612198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bb6b9-c0f9-4a10-b29c-ce66e96559cc None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.612678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bb6b9-c0f9-4a10-b29c-ce66e96559cc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.626939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95bb6b9-c0f9-4a10-b29c-ce66e96559cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:33.632079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:33.633168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-f95bb6b9-c0f9-4a10-b29c-ce66e96559cc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:33.633526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:33.635281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:33.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:33.638233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.638233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.640099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 183/375] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:33.641289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bb6b9-c0f9-4a10-b29c-ce66e96559cc None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:33.649986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:33.652743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:33.679319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:33.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:33.681956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 193/376] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Oct 19 00:27:33 2022] GET /identity/v3/users/nova => generated 82 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:33.683101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:33.734361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:33.769966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:33.785177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bb6b9-c0f9-4a10-b29c-ce66e96559cc None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:33.832072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:33.832918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.833184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a7d933-c58c-4f1e-b0c1-daa101ded4f1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:33.833353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.833777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.834895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] RBAC: Adding query filter params (name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:33.835381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:33.837411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:33.850177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 184/377] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:33.852301 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:33.888229 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5c868a-17dc-4c06-ae90-7613bda34eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:33.888950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5c868a-17dc-4c06-ae90-7613bda34eb2 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.889327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5c868a-17dc-4c06-ae90-7613bda34eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.890238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5c868a-17dc-4c06-ae90-7613bda34eb2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.905250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5c868a-17dc-4c06-ae90-7613bda34eb2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-2a5c868a-17dc-4c06-ae90-7613bda34eb2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:33.914623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:33.915864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 194/378] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:33.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dd91a-a6c6-47b3-a737-33bb608b8f92 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:33.920057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dd91a-a6c6-47b3-a737-33bb608b8f92 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:33.920057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dd91a-a6c6-47b3-a737-33bb608b8f92 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:33.920057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 195/379] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:33.922090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812dd91a-a6c6-47b3-a737-33bb608b8f92 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:33.997894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c70722-9435-4a37-a5ae-eb9739c55b4e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:34.033205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:34.044032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:34.044766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.045162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.045568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.046375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] RBAC: Adding query filter params (name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:34.046848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:34.048489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:34.049769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812dd91a-a6c6-47b3-a737-33bb608b8f92 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:34.058327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 185/380] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:34.059480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:34.069602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:34.099518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:34.100384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.100785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.101149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.102308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:34.102822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:34.105359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:34.170355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 196/381] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Oct 19 00:27:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:34.171666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:34.199104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c8e95-0929-41e1-8b63-b882a4456289 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:34.239455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c5bf8a-3067-4d13-bd23-0c9848d3cc1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:34.240199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5bf8a-3067-4d13-bd23-0c9848d3cc1d None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.240544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5bf8a-3067-4d13-bd23-0c9848d3cc1d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.240899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5bf8a-3067-4d13-bd23-0c9848d3cc1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.253587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c5bf8a-3067-4d13-bd23-0c9848d3cc1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:34.255094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c5bf8a-3067-4d13-bd23-0c9848d3cc1d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-74c5bf8a-3067-4d13-bd23-0c9848d3cc1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:34.269949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:34.271831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:34.273806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:34.273806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:34.273806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:34.273806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 186/382] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:34.273806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95590efb-be73-42c6-81b7-9615b8a7659f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.273806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95590efb-be73-42c6-81b7-9615b8a7659f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.273806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95590efb-be73-42c6-81b7-9615b8a7659f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.292547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-95590efb-be73-42c6-81b7-9615b8a7659f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:34.293760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-95590efb-be73-42c6-81b7-9615b8a7659f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:34.304684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebab14ee-6090-4787-b1cd-e54f66bfff61 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:34.331881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:34.343370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:34.344345 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.344704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.345137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.346040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] RBAC: Adding query filter params (name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:34.346429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:34.348201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:34.356741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 197/383] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:34.359655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:34.388878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 187/384] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:34.391686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95590efb-be73-42c6-81b7-9615b8a7659f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:34.393971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:34.484340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fda3172-bf9c-4dd7-ad5f-b0ba0ae9f047 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:34.542439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:34.542791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95590efb-be73-42c6-81b7-9615b8a7659f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:34.543128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.543412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.543687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.559759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:34.568494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:34.593476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:34.594112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.594383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.594702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:34.597034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:34.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:34.602329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:34.602329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:34.602329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:34.602329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 198/385] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:34.602329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:34.614243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:34.615713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:34.633332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:34.635160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:34.637143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:34.637143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:34.637143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:34.637143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 188/386] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:34.638547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:34.673702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:34.761471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:34.783384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52321f53-acd2-41d8-a399-fcf637a5a375 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:34.788511 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba33a8f0-489e-423f-bbea-52da0d99a521 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:34.825803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:34.826619 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.827974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.828371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.828743 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:34.829241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:34.829505 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:34.829708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:34.829877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:34.830239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:34.831404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:34.843824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:34.845395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 199/387] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:34.845580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:34.846822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:34.863486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:34.865592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:34.867626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:34.867626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:34.867626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:34.867626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 189/388] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:34.867626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:34.971955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:34.994760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c604d9e-65e5-4e4d-861e-cb4687958170 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.001254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8192e3c8-b83b-4e74-bbc3-442d79dd63e9 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.043461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.044240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.044581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.044919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.045660 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:35.046096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.047938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.049284 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.050331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.050863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.051344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.052557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:35.053091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.055650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.064415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 190/389] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:35.065303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.076143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 200/390] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:35.078147 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.082941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:35.210715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d189ddf0-f0b2-438f-a1c1-7b768f296d72 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.240274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c972f0ed-9759-49e3-8745-78732d889a79 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.262062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.262827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.263186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.263559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.266360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:35.280465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.282416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.284819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.285903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.286375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.286889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.297566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.299087 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:35.300475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:35.302374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:35.304345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:35.304345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:35.304345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:35.304345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 191/391] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:35.304345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:35.319914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:35.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:35.322827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 201/392] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/users/cinder => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:35.323420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.398333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:35.443169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bffd7a-e782-4dec-bac5-18333eae22eb None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.457217 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2424077-8878-494d-9699-d7d70ab0da7d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.479081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.479610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.479890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.480163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.480814 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:35.481116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.482509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.498021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 192/393] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:35.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.516616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.517392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.517745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.518115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.526064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.527606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:35.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:35.549473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 202/394] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:35.551279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.579575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:35.643735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a89adff-e180-4928-9794-3b224a181f85 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.689006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1919c3f-d30c-48ef-8d41-b8ac6506771d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.689582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.690289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.690687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.691023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] PATH_INFO: `/v3/users/glance` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.699695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.701570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:35.721928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:35.724124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 193/395] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/users/glance => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:35.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.728970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:35.751783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.752572 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.752897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.754052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:35.754413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.756004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.770302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 203/396] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:35.771975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.873902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de59f47-954d-4c88-a8c2-71d3148b3f8a None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.883731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:35.909934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df27821-84bd-4b8a-8e4b-d861f838c0dd None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:35.922533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.923160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.923439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.923710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.924476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:35.924783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.926303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.956880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:35.957686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:35.958044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:35.958413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:35.968091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:35.969729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:35.970078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 194/397] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:35.970958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:35.986835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:35.987854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 204/398] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/users/neutron => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:35.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:36.043095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:36.105293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c1361-c425-42ec-867b-5eddc3747af5 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:36.126335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfa1d38-314e-4307-9913-536007b1917a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:36.146914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:36.147712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:36.148075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:36.148454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:36.149396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:36.149748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:36.151500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:36.172526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 195/399] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:36.173224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:36.191631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:36.192414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:36.192886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:36.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:36.194774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:36.196467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:36.212175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:36.231059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 205/400] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Oct 19 00:27:35 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:36.231059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:36.322626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144c9f18-1c97-4128-a1ac-6c6db9a1a803 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:36.360863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cc45cb-4e48-4c50-9121-037f543db824 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:36.362849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:36.376556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb77fa1-1672-4967-afc3-e7ad75a9f9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:36.377181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb77fa1-1672-4967-afc3-e7ad75a9f9bd None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:36.377463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb77fa1-1672-4967-afc3-e7ad75a9f9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:36.377735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb77fa1-1672-4967-afc3-e7ad75a9f9bd None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:36.378394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb77fa1-1672-4967-afc3-e7ad75a9f9bd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:36.379914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb77fa1-1672-4967-afc3-e7ad75a9f9bd None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:36.403144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 196/401] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:27:36 2022] POST /identity/v3/users => generated 247 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:36.404287 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb77fa1-1672-4967-afc3-e7ad75a9f9bd None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:36.407809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:36.408472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:36.408753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:36.409028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:36.429013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:36.432006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:36.459715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:36.460709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:36.461555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:36.461555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:36.461555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:36.461555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 206/402] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:36.464510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:36.530535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:36.568720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb77fa1-1672-4967-afc3-e7ad75a9f9bd None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:36.621059 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:36.621610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:36.621891 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:36.622161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:36.622988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:36.623303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:36.625537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:36.649930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b47de-7bbf-4fc9-8272-52de77653036 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:36.652251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 197/403] 10.222.0.26 () {56 vars in 1139 bytes} [Wed Oct 19 00:27:36 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:36.653463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:36.697452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7680cb8f-681c-4245-9a7a-a95766984ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:36.698151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7680cb8f-681c-4245-9a7a-a95766984ec2 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:36.698434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7680cb8f-681c-4245-9a7a-a95766984ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:36.698739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7680cb8f-681c-4245-9a7a-a95766984ec2 None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/41009aec3b1843aa80323c374cc01461/roles/8b6c794a10cf4fcdb33edf34796d4819` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:36.719393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7680cb8f-681c-4245-9a7a-a95766984ec2 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=41009aec3b1843aa80323c374cc01461, role_id=8b6c794a10cf4fcdb33edf34796d4819)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:36.720926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7680cb8f-681c-4245-9a7a-a95766984ec2 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:36.740186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 207/404] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:36 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/41009aec3b1843aa80323c374cc01461/roles/8b6c794a10cf4fcdb33edf34796d4819 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:36.741049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7680cb8f-681c-4245-9a7a-a95766984ec2 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:36.746887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:36.836389 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d52bf4-ecab-48e8-b531-c072037d8bd1 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:36.912728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:36.915245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:36.916149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:36.916644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:36.928024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:36.941220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:36.943457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:36.968456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7680cb8f-681c-4245-9a7a-a95766984ec2 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:36.969331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:36.970225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:36.971124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:36.971124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:36.971124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:36.972261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 198/405] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:36.978384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:37.043515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:37.044198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:37.044477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:37.044752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:37.046032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:37.046342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:37.048735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:37.082511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 208/406] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:37.082511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:37.194619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ab4b85-b096-4b63-8d31-00079b1abad5 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:37.234737 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:37.259157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:37.259827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:37.262517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:37.262517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:37.289025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:37.293562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:37.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:37.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:37.328531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:37.328531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:37.328531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:37.328531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 199/407] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 355 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:37.333115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:37.337657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9cfb47-81b7-40e7-b2ab-48e48100175c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:37.421182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:37.421955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:37.422548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:37.422830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:37.423764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:37.424070 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:37.425641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:37.432367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:37.445016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 209/408] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:37.445878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:37.530376 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f44f81e-e9ac-4d19-987c-1e1ac6ab3dc9 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:37.589602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71595b82-7a8f-4138-8f95-afefa12458ea None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:37.592836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:37.594878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:37.595485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:37.595964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:37.606325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:37.608632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:37.621632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:37.630638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:37.632449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 200/409] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:37.634159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:37.648811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:37.650126 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:37.650849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:37.651827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:37.653411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:37.654202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:37.656586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:37.680046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 210/410] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:37.681253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:37.752303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc35b59d-3643-4f17-8141-55e78698b276 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:37.756133 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:37.791740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:37.792551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:37.792934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:37.793312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:37.801527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:37.803013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:37.815294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:37.816408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:37.817490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:37.817490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:37.817490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:37.817490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:37.817490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:37.817490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:37.817490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:37.817490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:37.819153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 201/411] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:37.820027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:37.821136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6a6422-942a-414e-a7cf-329feda78158 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:37.868162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:37.869276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:37.869728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:37.870165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:37.871127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:37.871570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:37.873312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:37.897583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 211/412] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:37.898903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:37.931279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6021aad3-adb4-4177-b954-94835be95e45 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:37.954712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:37.968075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:37.968915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:37.969237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:37.969603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:37.977428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:37.979037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:37.992003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:37.993011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:37.993847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:37.993847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:37.993847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:37.993847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:37.993847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:37.993847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:37.993847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:37.993847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:37.999270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 202/413] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:38.000416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:38.025300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88020aca-8541-4f28-842b-92ad19965144 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:38.025821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:38.070314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:38.071709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:38.072168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:38.072725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:38.073588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:38.074069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:38.075977 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:38.091979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 212/414] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:38.094177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:38.113219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76f2e0-866d-4e36-aecb-69b702a83a99 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:38.161278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431798f6-6b7a-418c-b15f-9bbb9b0ed1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:38.162533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431798f6-6b7a-418c-b15f-9bbb9b0ed1e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:38.162996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431798f6-6b7a-418c-b15f-9bbb9b0ed1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:38.163928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431798f6-6b7a-418c-b15f-9bbb9b0ed1e0 None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/13270581a86e430886c878ff84302764/roles/8b6c794a10cf4fcdb33edf34796d4819` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:38.189210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431798f6-6b7a-418c-b15f-9bbb9b0ed1e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=13270581a86e430886c878ff84302764, role_id=8b6c794a10cf4fcdb33edf34796d4819)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:38.190692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431798f6-6b7a-418c-b15f-9bbb9b0ed1e0 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:38.207898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 203/415] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:37 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/13270581a86e430886c878ff84302764/roles/8b6c794a10cf4fcdb33edf34796d4819 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:38.209042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431798f6-6b7a-418c-b15f-9bbb9b0ed1e0 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:38.219647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07af8d31-7c56-4b55-80c8-502d4b07e623 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:38.223978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:38.261724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:38.262779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:38.263369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:38.263873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:38.265062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:38.265595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:38.267602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:38.282298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 213/416] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:38.288506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:38.303272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:38.380423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431798f6-6b7a-418c-b15f-9bbb9b0ed1e0 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:38.460487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34acd299-cd70-4143-93e3-6f2698b6cef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:38.461148 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34acd299-cd70-4143-93e3-6f2698b6cef8 None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:38.461429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34acd299-cd70-4143-93e3-6f2698b6cef8 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:38.461702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34acd299-cd70-4143-93e3-6f2698b6cef8 None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/7e3f6be584e34bfb9e430c7756d071d5/roles/8b6c794a10cf4fcdb33edf34796d4819` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:38.478773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c378680b-36d9-46a8-a888-de6deefef593 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:38.479288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34acd299-cd70-4143-93e3-6f2698b6cef8 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=7e3f6be584e34bfb9e430c7756d071d5, role_id=8b6c794a10cf4fcdb33edf34796d4819)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:38.480741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34acd299-cd70-4143-93e3-6f2698b6cef8 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:38.498975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 204/417] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:38 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/7e3f6be584e34bfb9e430c7756d071d5/roles/8b6c794a10cf4fcdb33edf34796d4819 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:38.532423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4375d687-a721-4f81-b13e-031889cb4985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:38.533491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375d687-a721-4f81-b13e-031889cb4985 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:38.533960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375d687-a721-4f81-b13e-031889cb4985 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:38.534420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375d687-a721-4f81-b13e-031889cb4985 None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/5756f4ff604a413b8af1f12c82cc4870/roles/8b6c794a10cf4fcdb33edf34796d4819` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:38.567458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375d687-a721-4f81-b13e-031889cb4985 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=5756f4ff604a413b8af1f12c82cc4870, role_id=8b6c794a10cf4fcdb33edf34796d4819)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:38.569375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375d687-a721-4f81-b13e-031889cb4985 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:38.589685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 214/418] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:38 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/5756f4ff604a413b8af1f12c82cc4870/roles/8b6c794a10cf4fcdb33edf34796d4819 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:27:38.979599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db685868-2278-452b-a6cc-1608815b5281 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:38.980633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db685868-2278-452b-a6cc-1608815b5281 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:38.981104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db685868-2278-452b-a6cc-1608815b5281 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:38.983100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 205/419] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:38.988500 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944bfb6-5d43-4edd-bae7-ffd621a76eff None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:38.988789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944bfb6-5d43-4edd-bae7-ffd621a76eff None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:38.989057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944bfb6-5d43-4edd-bae7-ffd621a76eff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:39.010135 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1944bfb6-5d43-4edd-bae7-ffd621a76eff None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:39.011439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1944bfb6-5d43-4edd-bae7-ffd621a76eff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:39.023077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:39.087956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 215/420] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:39.119685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93c7757-81d6-4542-b821-3db29c4648ac None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:39.120116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93c7757-81d6-4542-b821-3db29c4648ac None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:39.120494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93c7757-81d6-4542-b821-3db29c4648ac None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:39.122255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 206/421] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:39.127918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee75d204-db62-4d6c-9d5b-43cfa828b408 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:39.128210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee75d204-db62-4d6c-9d5b-43cfa828b408 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:39.128491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee75d204-db62-4d6c-9d5b-43cfa828b408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:39.151077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ee75d204-db62-4d6c-9d5b-43cfa828b408 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:39.152279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ee75d204-db62-4d6c-9d5b-43cfa828b408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:39.251631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 216/422] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:39.372839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93c7757-81d6-4542-b821-3db29c4648ac None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:39.506790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93c7757-81d6-4542-b821-3db29c4648ac None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:39.537560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:39.560352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b0913d-9312-48d8-96f2-2a1a149d16f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:39.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b0913d-9312-48d8-96f2-2a1a149d16f5 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:39.561837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b0913d-9312-48d8-96f2-2a1a149d16f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:39.562167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b0913d-9312-48d8-96f2-2a1a149d16f5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:39.573858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b0913d-9312-48d8-96f2-2a1a149d16f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-23b0913d-9312-48d8-96f2-2a1a149d16f5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:39.579509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:39.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:39.581270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 207/423] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:39.586899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee75d204-db62-4d6c-9d5b-43cfa828b408 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:39.632255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b0913d-9312-48d8-96f2-2a1a149d16f5 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:39.723170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:39.758444 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee75d204-db62-4d6c-9d5b-43cfa828b408 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:39.833103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:39.833977 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:39.834262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:39.834559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:39.836085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] RBAC: Adding query filter params (name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:39.836394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:39.837789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:39.848927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b0913d-9312-48d8-96f2-2a1a149d16f5 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:39.859759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 217/424] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:39.866794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:39.916579 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13371ac1-62c7-4df3-93a9-ba1ffe089f69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:39.917217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13371ac1-62c7-4df3-93a9-ba1ffe089f69 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:39.917497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13371ac1-62c7-4df3-93a9-ba1ffe089f69 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:39.917771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13371ac1-62c7-4df3-93a9-ba1ffe089f69 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:39.940631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13371ac1-62c7-4df3-93a9-ba1ffe089f69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-13371ac1-62c7-4df3-93a9-ba1ffe089f69 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:39.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:39.948246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:39.949068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 208/425] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:39.954631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13371ac1-62c7-4df3-93a9-ba1ffe089f69 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:40.007517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:40.025472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca24d52-2049-44fd-b9bb-6ba22785063c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:40.072212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:40.072872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:40.073156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:40.073430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:40.081140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:40.098713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:40.100212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:40.102526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13371ac1-62c7-4df3-93a9-ba1ffe089f69 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:40.122110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:40.123123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:40.124573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:40.124573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:40.124573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:40.124573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 218/426] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:40.135488 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:40.143197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:40.144065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:40.144491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:40.144933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:40.145739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] RBAC: Adding query filter params (name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:40.146184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:40.147800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:40.165314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 209/427] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:40.171755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:40.245600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cf3de1-661f-433f-8973-15ee54fe9eac None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:40.285034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:40.285571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:40.285850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:40.286123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:40.286797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:40.287100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:40.288468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:40.304376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 219/428] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:40.312330 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:40.382965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942a53ec-1ff9-4217-997d-6832eaddfa3c None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:40.391186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:40.414094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:40.454518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:40.455632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:40.457283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:40.457705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:40.473203 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:40.474815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:40.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:40.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:40.515049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:40.515049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:40.515049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:40.518740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 210/429] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:40.524300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:40.528149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d439a16-dcfd-4f34-9807-af4b1f388fe5 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:40.608895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:40.609555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:40.609843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:40.610269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] PATH_INFO: `/v3/users/swift` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:40.618434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:40.619954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:40.632614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:40.633561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:40.634442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 220/430] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:27:40 2022] GET /identity/v3/users/swift => generated 83 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:40.640545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:40.735759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffa3757-5d1e-4fd8-8ee5-99c6d03a619f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:40.784796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92755c0-533f-4a6a-ac5e-e69191c1c1db None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:40.794745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:40.824770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:40.825456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:40.825809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:40.826200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:40.827438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:40.827750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:40.827998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:40.828661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:40.828976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:40.829274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:40.829552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:40.829967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:40.830293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:40.831835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:40.849943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 221/431] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Oct 19 00:27:40 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:40.861106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 211/432] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:40.862330 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:40.866398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:40.890117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:41.002375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b11c08-7f75-4c0e-bd08-21d40390432f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:41.054757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d5fd72-ec7b-453f-acc6-5959960c5219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:41.056219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d5fd72-ec7b-453f-acc6-5959960c5219 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.056739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d5fd72-ec7b-453f-acc6-5959960c5219 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.057201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d5fd72-ec7b-453f-acc6-5959960c5219 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.066695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d5fd72-ec7b-453f-acc6-5959960c5219 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:41.068611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d5fd72-ec7b-453f-acc6-5959960c5219 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:41.068970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6059a5-c6d7-4b86-a31c-68524fef9216 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-28d5fd72-ec7b-453f-acc6-5959960c5219 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:41.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:41.084375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:27:41.085666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:41.088031 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 212/433] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Oct 19 00:27:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:41.089561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9101637-95f2-49b0-b1c2-7b7fdf3eec93 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.089999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9101637-95f2-49b0-b1c2-7b7fdf3eec93 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.090420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9101637-95f2-49b0-b1c2-7b7fdf3eec93 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.091960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 213/434] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:41.093945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9101637-95f2-49b0-b1c2-7b7fdf3eec93 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:41.119017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1dc666-8f5c-41e9-a272-df6b87173dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:41.119833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1dc666-8f5c-41e9-a272-df6b87173dae None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.120202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1dc666-8f5c-41e9-a272-df6b87173dae None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.120568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1dc666-8f5c-41e9-a272-df6b87173dae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.131966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1dc666-8f5c-41e9-a272-df6b87173dae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:41.133544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1dc666-8f5c-41e9-a272-df6b87173dae None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-de1dc666-8f5c-41e9-a272-df6b87173dae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:41.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:41.158783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:41.159994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:41.159994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:41.159994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:41.160971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 222/435] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:41.166600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d2cdd1-6c12-4117-a69e-7d8dba2e6ed2 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.166600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d2cdd1-6c12-4117-a69e-7d8dba2e6ed2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.166600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d2cdd1-6c12-4117-a69e-7d8dba2e6ed2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.188456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a6d2cdd1-6c12-4117-a69e-7d8dba2e6ed2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:41.190961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a6d2cdd1-6c12-4117-a69e-7d8dba2e6ed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:41.201631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:41.207596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:41.238649 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9101637-95f2-49b0-b1c2-7b7fdf3eec93 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:41.277811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:41.278734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.279149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.280194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.281102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:41.281546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:41.283219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:41.284551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 223/436] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:41.285682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d2cdd1-6c12-4117-a69e-7d8dba2e6ed2 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:41.321596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 214/437] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Oct 19 00:27:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:41.326236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:41.461720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d2cdd1-6c12-4117-a69e-7d8dba2e6ed2 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:41.471401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d484266-8759-4cda-ae9c-ba882f1a7c42 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:41.497260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4606a37-1ba6-4bce-8826-598d7f606f3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:41.497876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4606a37-1ba6-4bce-8826-598d7f606f3e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.498156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4606a37-1ba6-4bce-8826-598d7f606f3e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.498512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4606a37-1ba6-4bce-8826-598d7f606f3e None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.499153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4606a37-1ba6-4bce-8826-598d7f606f3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:41.499454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4606a37-1ba6-4bce-8826-598d7f606f3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:41.500833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4606a37-1ba6-4bce-8826-598d7f606f3e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:41.508931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a12d70-e6b3-4d27-b32f-cf9653095a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:41.509663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a12d70-e6b3-4d27-b32f-cf9653095a3b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.509944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a12d70-e6b3-4d27-b32f-cf9653095a3b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.510215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a12d70-e6b3-4d27-b32f-cf9653095a3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.513102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 224/438] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:41.513264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:41.515556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4b7b68-8282-414d-a76b-4154233cc403 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.515997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4b7b68-8282-414d-a76b-4154233cc403 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.516370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4b7b68-8282-414d-a76b-4154233cc403 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.522199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 225/439] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:41.522199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a36d35e-7332-4af8-912e-ea8c297e37af None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.522199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a36d35e-7332-4af8-912e-ea8c297e37af None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.522199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a36d35e-7332-4af8-912e-ea8c297e37af None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.522199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 226/440] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:41 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:41.525698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a36d35e-7332-4af8-912e-ea8c297e37af None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:41.538562 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:41.593980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a12d70-e6b3-4d27-b32f-cf9653095a3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:41.595471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a12d70-e6b3-4d27-b32f-cf9653095a3b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:41.646291 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a36d35e-7332-4af8-912e-ea8c297e37af None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:41.682547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3025c7f-690a-4acd-b19b-42315eeaa1df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:41.683169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3025c7f-690a-4acd-b19b-42315eeaa1df None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.683446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3025c7f-690a-4acd-b19b-42315eeaa1df None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.683717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3025c7f-690a-4acd-b19b-42315eeaa1df None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.690594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3025c7f-690a-4acd-b19b-42315eeaa1df None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:41.692034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3025c7f-690a-4acd-b19b-42315eeaa1df None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:41.698418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 215/441] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 19 00:27:41.698418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883c2e33-4224-48b1-b68a-e190bf2627b1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.698418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883c2e33-4224-48b1-b68a-e190bf2627b1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.698418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883c2e33-4224-48b1-b68a-e190bf2627b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-a3025c7f-690a-4acd-b19b-42315eeaa1df None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:41.705258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:41.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:41.707216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 227/442] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:41.707919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfc265f-6a37-4f13-b656-1c3392602999 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.707919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfc265f-6a37-4f13-b656-1c3392602999 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.708598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfc265f-6a37-4f13-b656-1c3392602999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.714080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-883c2e33-4224-48b1-b68a-e190bf2627b1 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:41.715243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-883c2e33-4224-48b1-b68a-e190bf2627b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:41.728029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ccfc265f-6a37-4f13-b656-1c3392602999 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:41.729231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ccfc265f-6a37-4f13-b656-1c3392602999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:41.744486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:41.788354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 216/443] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:41.795653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883c2e33-4224-48b1-b68a-e190bf2627b1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:41.799992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:41.820933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 228/444] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:41.820933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfc265f-6a37-4f13-b656-1c3392602999 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:41.912309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883c2e33-4224-48b1-b68a-e190bf2627b1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:41.948503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfc265f-6a37-4f13-b656-1c3392602999 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:41.958258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325e3bb7-9eeb-4b13-bbd8-d7c06c3095cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:41.959138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325e3bb7-9eeb-4b13-bbd8-d7c06c3095cc None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.959577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325e3bb7-9eeb-4b13-bbd8-d7c06c3095cc None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.959986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325e3bb7-9eeb-4b13-bbd8-d7c06c3095cc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:41.975408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325e3bb7-9eeb-4b13-bbd8-d7c06c3095cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-325e3bb7-9eeb-4b13-bbd8-d7c06c3095cc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:41.982468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:41.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:41.985253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:41.992721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 217/445] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:41.992721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325e3bb7-9eeb-4b13-bbd8-d7c06c3095cc None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:41.997525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:41.999003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:41.999003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:41.999003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.012294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.014380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.026680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:42.034582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:42.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:42.036781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:42.036781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:42.036781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:42.038257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 229/446] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:42.039249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.101475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:42.114258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325e3bb7-9eeb-4b13-bbd8-d7c06c3095cc None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.152250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.152891 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.153199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.153473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.154217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:42.154545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.156671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.160580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccee9f-725e-448a-b660-2dabce11bc29 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.168911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 218/447] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:42.169731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.197530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.197811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.198088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.198789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] RBAC: Adding query filter params (name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:42.199130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.200474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.209250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 230/448] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:42.210067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.255824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:42.272566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a154669-ab6b-4f43-829d-f72c9218ddd4 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.306351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.306983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.307265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.307536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.308171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:42.308474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.309845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.318151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b15984-2f37-434d-a190-5e5c433c4734 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.322035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 219/449] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:42.327069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.333074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:42.358438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e420da7-35a9-40a1-8de6-bc2cc6c3aae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.359796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e420da7-35a9-40a1-8de6-bc2cc6c3aae3 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.360239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e420da7-35a9-40a1-8de6-bc2cc6c3aae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.360516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e420da7-35a9-40a1-8de6-bc2cc6c3aae3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.372638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e420da7-35a9-40a1-8de6-bc2cc6c3aae3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-6e420da7-35a9-40a1-8de6-bc2cc6c3aae3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:42.378820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:42.379983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 231/450] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:42.380781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e420da7-35a9-40a1-8de6-bc2cc6c3aae3 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.435546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db736bcc-37ce-44f4-85ed-593396ec5c75 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.473186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca72305-2a25-4e79-9d40-4415e7400769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.473810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca72305-2a25-4e79-9d40-4415e7400769 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.474085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca72305-2a25-4e79-9d40-4415e7400769 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.474353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca72305-2a25-4e79-9d40-4415e7400769 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.485483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca72305-2a25-4e79-9d40-4415e7400769 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:42.487846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e420da7-35a9-40a1-8de6-bc2cc6c3aae3 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-2ca72305-2a25-4e79-9d40-4415e7400769 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:42.490330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:42.491406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 220/451] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:42.492309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca72305-2a25-4e79-9d40-4415e7400769 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.497166 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:42.524629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c929bf49-210f-481b-a238-1d236552ca29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.525202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c929bf49-210f-481b-a238-1d236552ca29 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.525480 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c929bf49-210f-481b-a238-1d236552ca29 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.525752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c929bf49-210f-481b-a238-1d236552ca29 None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/33b9c777f8284edeb758b00de044fd45/roles/8b6c794a10cf4fcdb33edf34796d4819` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.542027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c929bf49-210f-481b-a238-1d236552ca29 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=33b9c777f8284edeb758b00de044fd45, role_id=8b6c794a10cf4fcdb33edf34796d4819)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.543805 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c929bf49-210f-481b-a238-1d236552ca29 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.559324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 232/452] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:42 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/33b9c777f8284edeb758b00de044fd45/roles/8b6c794a10cf4fcdb33edf34796d4819 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:42.562122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c929bf49-210f-481b-a238-1d236552ca29 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.594243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:42.618101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca72305-2a25-4e79-9d40-4415e7400769 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.662593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.663244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.663643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.663923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.679171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.680775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.695304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c929bf49-210f-481b-a238-1d236552ca29 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:42.697571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:42.698610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:42.699472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:42.699472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:42.699472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:42.699472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 221/453] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:42.699703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.745617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.746862 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.747156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.747429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.757479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.758371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.771074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:42.772699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:42.773547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:42.774402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 233/454] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:42.774992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.829114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ff034f-5406-4ce3-ab66-1e9072f892bb None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.870054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.871120 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.871398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.871666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.872675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] RBAC: Adding query filter params (name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:42.872960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.875207 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.876570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:42.889838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 222/455] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:42.889838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:42.905589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4161e23-37f8-4dee-90f1-54b0b68da864 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:42.951082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:42.952454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:42.952854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:42.953137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:42.953827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:42.954138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:42.955592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:42.972144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 234/456] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:42.972144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.000572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7f0292-0b10-4a70-a90a-3a2142c89e1f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.018892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:43.038852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.039379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.039656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.039923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.040560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:43.040862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.042243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.062464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 223/457] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:43.062464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.119493 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d7860f-3444-4938-9cc9-0bd24c82dcc0 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.164633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.165460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.165770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.166093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.166916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:43.167310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.169125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.180961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253a51dc-b960-4c0e-99b9-7f5c008d9019 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.184773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 235/458] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:43.185763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.196535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:43.229519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.230287 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.230673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.231025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.244110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.245734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:43.261898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:43.262925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:43.264187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:43.264187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:43.264187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:43.264187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 224/459] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:43.265063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.306522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:43.340222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bedeec8-2339-46fd-9f27-90972bece092 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.386627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.387446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.387917 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.388348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.389090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9b5d0e-4e4b-49cf-9eba-4fc38b05c52f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.402894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.404385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:43.424700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:43.425898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:43.431692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:43.431692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:43.431692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:43.431692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 236/460] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:43.431692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.436278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.437025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.437222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.437495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.447530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.449404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:43.462103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:43.463725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 00:27:43.466165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:43.468825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 225/461] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/users/cinder => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:43.469832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.509381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:43.579525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbcdbc6-9ee9-4312-8950-5987334fbd2d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.593334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:43.607827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b717e3-1b32-4f54-9627-cc134ff5dadc None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.625016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.626014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.626509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.626973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.627888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] RBAC: Adding query filter params (role.id=8b6c794a10cf4fcdb33edf34796d4819, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=41009aec3b1843aa80323c374cc01461) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:43.628419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.630115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.636607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 237/462] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/role_assignments?user.id=41009aec3b1843aa80323c374cc01461&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=8b6c794a10cf4fcdb33edf34796d4819 => generated 608 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:43.637683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.661346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.662376 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.662841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.663149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.663833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:43.664137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.665609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.686896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 226/463] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:43.687893 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.770736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71402d20-c064-49b2-91c2-ca1bfe364575 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.797607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:43.820608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.821278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.821610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.821886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.822748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:43.823189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.824624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.846643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 238/464] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:43.848161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.852011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f340348-139d-44b9-9d3b-1c127ef6ad85 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:43.898099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:43.898763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:43.899045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:43.899316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:43.900061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:43.900366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:43.901847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:43.925999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 227/465] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:43.927151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:43.939312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:44.001580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199bda8a-6122-4658-a274-f2a713d8df96 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.048925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.049753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.050164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.050648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.053171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4145d-7b69-42be-9347-0059ced02a69 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.066287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.069292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:44.087067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:44.088059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 00:27:44.088988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:44.093257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 239/466] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/users/neutron => generated 85 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:44.094275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:44.100109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.100834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.101204 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.101566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.112977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.114739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:44.128782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:44.129967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 19 00:27:44.131001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:44.137623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 228/467] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:43 2022] GET /identity/v3/users/glance => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:44.138889 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:44.145950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:44.215119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932c612a-c653-464b-81b2-67699b934f4c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.220331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:44.256837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60db5539-e52b-4a0f-bbc8-653cdc14d505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.257467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60db5539-e52b-4a0f-bbc8-653cdc14d505 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.257749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60db5539-e52b-4a0f-bbc8-653cdc14d505 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.258026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60db5539-e52b-4a0f-bbc8-653cdc14d505 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.262683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc91f95-6b88-4580-96fb-6649d0549581 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.301736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.302682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.303101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.303508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.304394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:44.304841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.306453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:44.327862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 229/468] 10.222.0.26 () {56 vars in 1139 bytes} [Wed Oct 19 00:27:44 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:44.329010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:44.340713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60db5539-e52b-4a0f-bbc8-653cdc14d505 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.342867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60db5539-e52b-4a0f-bbc8-653cdc14d505 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:44.392034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:44.447871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4c3c5a-257d-426d-a8f7-fa4d481b5a5f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.460513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 240/469] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:44.461409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60db5539-e52b-4a0f-bbc8-653cdc14d505 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:44.481745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:44.494818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.495801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.496414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.496902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.497907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:44.498961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.500648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:44.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 230/470] 10.222.0.26 () {56 vars in 1137 bytes} [Wed Oct 19 00:27:44 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:44.525587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:44.581748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60db5539-e52b-4a0f-bbc8-653cdc14d505 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.620528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c1661-9a4d-4363-8075-bbf04d107615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.621202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c1661-9a4d-4363-8075-bbf04d107615 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.621430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c1661-9a4d-4363-8075-bbf04d107615 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.621702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c1661-9a4d-4363-8075-bbf04d107615 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.633935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:44.650130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdd5c2-4c29-4b8c-8065-93a05800e245 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.704898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.705937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.706536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.707039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.710263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3c1661-9a4d-4363-8075-bbf04d107615 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.711777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3c1661-9a4d-4363-8075-bbf04d107615 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:44.720125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.722151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:44.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:44.742633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:44.743503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:44.743503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:44.743503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:44.745951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 231/471] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:44.747115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:44.789155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:44.836534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 241/472] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:44.837693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c1661-9a4d-4363-8075-bbf04d107615 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:44.861426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b14ec5-5e9d-4a8d-ac69-4cc69761f1a8 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.888590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:44.899687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e8c86-0a13-4854-9148-3a84e374819e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.900355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e8c86-0a13-4854-9148-3a84e374819e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.900576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e8c86-0a13-4854-9148-3a84e374819e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.900843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e8c86-0a13-4854-9148-3a84e374819e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.955858 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c1661-9a4d-4363-8075-bbf04d107615 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:44.975938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e8c86-0a13-4854-9148-3a84e374819e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.977393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e8c86-0a13-4854-9148-3a84e374819e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:44.993356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516c6982-aee2-40b0-ac10-381f9a31c706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:44.993999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c6982-aee2-40b0-ac10-381f9a31c706 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:44.994254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c6982-aee2-40b0-ac10-381f9a31c706 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:44.994548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c6982-aee2-40b0-ac10-381f9a31c706 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:44.995236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516c6982-aee2-40b0-ac10-381f9a31c706 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:44.995496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516c6982-aee2-40b0-ac10-381f9a31c706 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:44.997099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516c6982-aee2-40b0-ac10-381f9a31c706 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:45.017042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 242/473] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:45.017042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec385c6-535e-4bb6-bc67-b7fc538a2bef None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:45.017042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec385c6-535e-4bb6-bc67-b7fc538a2bef None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:45.017042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec385c6-535e-4bb6-bc67-b7fc538a2bef None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:45.017042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 243/474] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:45.017042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec385c6-535e-4bb6-bc67-b7fc538a2bef None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:45.023781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:45.102472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 232/475] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:45.104302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e8c86-0a13-4854-9148-3a84e374819e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:45.132442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:45.137304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec385c6-535e-4bb6-bc67-b7fc538a2bef None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:45.185331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19ffaa7-89c3-41bd-8b3e-80e1093a1293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:45.185926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19ffaa7-89c3-41bd-8b3e-80e1093a1293 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:45.186203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19ffaa7-89c3-41bd-8b3e-80e1093a1293 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:45.186470 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19ffaa7-89c3-41bd-8b3e-80e1093a1293 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:45.200592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19ffaa7-89c3-41bd-8b3e-80e1093a1293 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:45.203874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19ffaa7-89c3-41bd-8b3e-80e1093a1293 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-e19ffaa7-89c3-41bd-8b3e-80e1093a1293 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:45.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:45.237397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:45.239535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:45.239535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:45.239535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:45.239535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 244/476] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:45.239535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a22fc-cff6-4e3d-bd7b-83d15f6a8c2c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:45.239535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a22fc-cff6-4e3d-bd7b-83d15f6a8c2c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:45.239535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104a22fc-cff6-4e3d-bd7b-83d15f6a8c2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:45.254298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-104a22fc-cff6-4e3d-bd7b-83d15f6a8c2c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:45.255535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-104a22fc-cff6-4e3d-bd7b-83d15f6a8c2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:45.259315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e8c86-0a13-4854-9148-3a84e374819e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:45.304122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:45.304759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:45.305035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:45.305313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:45.308939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:45.314328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:45.315923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:45.329456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:45.331191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 233/477] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:45.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:45.358518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 245/478] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:45.358518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a22fc-cff6-4e3d-bd7b-83d15f6a8c2c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:45.472145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc26a4-097f-41a8-a177-e450942bad3f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:45.512543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:45.513028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:45.513313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:45.513567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:45.526957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:45.528430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:45.554206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:45.555826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:45.557678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:45.557678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:45.557678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:45.557678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 234/479] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:45.557678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:45.574180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:45.599003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104a22fc-cff6-4e3d-bd7b-83d15f6a8c2c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:45.632458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:45.691394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:45.692021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:45.692297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:45.692569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:45.693257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:45.693557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:45.694985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:45.707735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1c1606-97ef-4a0a-8b9e-90d046c99cc1 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:45.714517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 246/480] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:45.714517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:45.798975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:45.800784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:45.801806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:45.803163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:45.803600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:45.806747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:45.810559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:45.838837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c91f4d-36c2-4b54-aedf-a69828669963 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:45.846756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 235/481] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:45.848436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:45.880423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:45.881057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:45.881495 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:45.881770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:45.882414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:45.882747 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:45.884122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:45.901351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 247/482] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:45.901351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:45.902984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:45.993639 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a922a2e-506a-4d8a-95f1-60cd16ca368f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.006841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:46.042256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9e7d9e-3a88-4c0a-9a71-31f18cff9a93 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.046919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.048017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.048510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.049054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.061076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:46.063747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:46.081365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ff0d41-2115-4017-991c-6cf4610ee97b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.082003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff0d41-2115-4017-991c-6cf4610ee97b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.082281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff0d41-2115-4017-991c-6cf4610ee97b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.082577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff0d41-2115-4017-991c-6cf4610ee97b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:46.086158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:46.087047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:46.087887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:46.087887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:46.087887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:46.087887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:46.087887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:46.087887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:46.087887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:46.087887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:46.088328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 236/483] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:46.089665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:46.100108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ff0d41-2115-4017-991c-6cf4610ee97b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-f3ff0d41-2115-4017-991c-6cf4610ee97b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:46.105558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:46.110721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 248/484] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:46.111542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ff0d41-2115-4017-991c-6cf4610ee97b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:46.191307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:46.231475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b6bde-30d2-4a3c-a708-59acc730add1 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.244122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ff0d41-2115-4017-991c-6cf4610ee97b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.272219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.272908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.273096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.273375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.274105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] RBAC: Adding query filter params (role.id=8b6c794a10cf4fcdb33edf34796d4819, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=13270581a86e430886c878ff84302764) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:46.274439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:46.275905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:46.281314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.281946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.282222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.282504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.282869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 237/485] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:46 2022] GET /identity/v3/role_assignments?user.id=13270581a86e430886c878ff84302764&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=8b6c794a10cf4fcdb33edf34796d4819 => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:46.283685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:46.290414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:46.291937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:46.305081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:46.315640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 249/486] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:46.317329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:46.342362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:46.447151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abe82c1-2ac7-4606-9b43-2dc0225d9904 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.455429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a18d8fb-5640-4a91-b769-c97ca7dc7e41 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.487002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:46.489888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74a84fe-7f3f-4374-ac1b-888b53a8aee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.490332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a84fe-7f3f-4374-ac1b-888b53a8aee2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.490658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a84fe-7f3f-4374-ac1b-888b53a8aee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.490927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a84fe-7f3f-4374-ac1b-888b53a8aee2 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.492399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74a84fe-7f3f-4374-ac1b-888b53a8aee2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:46.492399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74a84fe-7f3f-4374-ac1b-888b53a8aee2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:46.493398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74a84fe-7f3f-4374-ac1b-888b53a8aee2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:46.502179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4219eaf2-d34a-4e62-a992-51880e63fbd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.502836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4219eaf2-d34a-4e62-a992-51880e63fbd4 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.503117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4219eaf2-d34a-4e62-a992-51880e63fbd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.503390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4219eaf2-d34a-4e62-a992-51880e63fbd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.504063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4219eaf2-d34a-4e62-a992-51880e63fbd4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:46.504362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4219eaf2-d34a-4e62-a992-51880e63fbd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:46.505774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4219eaf2-d34a-4e62-a992-51880e63fbd4 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:46.511792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 238/487] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:46.512807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41922c5-44fb-4e71-81a9-2abd772b6b30 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.513098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41922c5-44fb-4e71-81a9-2abd772b6b30 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.515042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41922c5-44fb-4e71-81a9-2abd772b6b30 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.516522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 239/488] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:46.518962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41922c5-44fb-4e71-81a9-2abd772b6b30 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:46.522828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 250/489] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:46.524096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24011f6-0039-441c-ab91-0b4d2fa15b64 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.524381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24011f6-0039-441c-ab91-0b4d2fa15b64 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.524657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24011f6-0039-441c-ab91-0b4d2fa15b64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.544585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b24011f6-0039-441c-ab91-0b4d2fa15b64 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:46.545919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b24011f6-0039-441c-ab91-0b4d2fa15b64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:46.621575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:46.633682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 251/490] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:46.633682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24011f6-0039-441c-ab91-0b4d2fa15b64 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:46.637717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41922c5-44fb-4e71-81a9-2abd772b6b30 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.677783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.678391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.678696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.679007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.679876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:46.680180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:46.681853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:46.696798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 240/491] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:46.697726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:46.767596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24011f6-0039-441c-ab91-0b4d2fa15b64 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.804144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:46.814287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.814999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.815280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.816100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:46.816892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] RBAC: Adding query filter params (role.id=8b6c794a10cf4fcdb33edf34796d4819, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=5756f4ff604a413b8af1f12c82cc4870) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:46.817399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:46.819069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:46.828786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 252/492] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:46 2022] GET /identity/v3/role_assignments?user.id=5756f4ff604a413b8af1f12c82cc4870&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=8b6c794a10cf4fcdb33edf34796d4819 => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:46.829712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:46.936510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fb4a89-7acc-4c20-8780-edbb9b70d057 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:46.982472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:46.984211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:46.984704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:46.985176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:47.000734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:47.002283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:47.022535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:47.024283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:47.027498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:47.027498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:47.027498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:47.028746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 241/493] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:47.030473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:47.036154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:47.083817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae1189a-a2ee-4cc4-b475-16d2c6999dfa None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:47.165137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:47.166749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:47.167542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:47.168310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:47.169982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] RBAC: Adding query filter params (role.id=8b6c794a10cf4fcdb33edf34796d4819, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=7e3f6be584e34bfb9e430c7756d071d5) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:47.172025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:47.173621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a72f4-d226-462d-b38e-5a27176ae238 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:47.175464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:47.187968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 253/494] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:46 2022] GET /identity/v3/role_assignments?user.id=7e3f6be584e34bfb9e430c7756d071d5&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=8b6c794a10cf4fcdb33edf34796d4819 => generated 608 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:47.189917 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:47.239566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4970667-af9a-4a2d-84b2-49396cd81505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:47.240221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4970667-af9a-4a2d-84b2-49396cd81505 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:47.240505 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4970667-af9a-4a2d-84b2-49396cd81505 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:47.240775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4970667-af9a-4a2d-84b2-49396cd81505 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:47.260004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4970667-af9a-4a2d-84b2-49396cd81505 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-b4970667-af9a-4a2d-84b2-49396cd81505 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:47.273613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:47.276732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:47.278366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 242/495] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:27:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:47.282311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4970667-af9a-4a2d-84b2-49396cd81505 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:47.299023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:47.405611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:47.408962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c892d00-d74e-4961-8365-974e0fcfa03a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:47.484338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:47.488345 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:47.491165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:47.492128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:47.493676 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:47.500508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:47.503713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:47.538474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 254/496] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:47.543898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:47.549531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4970667-af9a-4a2d-84b2-49396cd81505 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:47.596630 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:47.597243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:47.597519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:47.597785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:47.598744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:47.599067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:47.600416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:47.611338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 243/497] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:47.616589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:47.768147 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c4382-7c45-4ced-8ca9-5edbbbbede93 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:47.774434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a66901-e61a-4150-b6f3-597662c49283 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:47.776247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:47.823309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:47.823309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:47.823309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:47.823309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:47.824367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:47.835579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:47.837063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:47.848911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:47.850297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:47.852175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:47.854065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:47.855932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 255/498] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:27:47 2022] GET /identity/v3/users/swift => generated 83 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:47.859045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:47.859624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:47.860604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:47.881360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:47.884595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:47.915901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:47.917691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:47.920289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:47.920289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:47.920289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:47.923740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 244/499] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:47.931612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:48.058203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e7680-c7d9-488b-91c7-18139880f362 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:48.135106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:48.135717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:48.135993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:48.136256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:48.137003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:48.137307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:48.140548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:48.161743 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e11d5d2-859c-49d0-9e5e-76f6087b56d0 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:48.183470 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 256/500] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Oct 19 00:27:47 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:48.192307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:48.246001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:48.247081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:48.248194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:48.249624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:48.250125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:48.251449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:48.252163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:48.255015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:48.286713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 245/501] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:48.294306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:48.439937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:48.476721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c751f405-5ed4-47bb-863f-0de387bafe5b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:48.590038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868402fd-a333-48cd-8563-8d64b1ec333d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:48.591152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868402fd-a333-48cd-8563-8d64b1ec333d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:48.591650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868402fd-a333-48cd-8563-8d64b1ec333d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:48.592850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868402fd-a333-48cd-8563-8d64b1ec333d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:48.635960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8cc39e-05e2-49d5-8ce1-3d2f94d19889 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:48.733913 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:48.735783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:48.737129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:48.738648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:48.757176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:48.760349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:48.779709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868402fd-a333-48cd-8563-8d64b1ec333d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:48.781198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868402fd-a333-48cd-8563-8d64b1ec333d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:48.790009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:48.791845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:48.793695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 246/502] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Oct 19 00:27:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 499 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:48.807851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:48.815891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:48.909362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 257/503] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 717 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 00:27:48.914620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868402fd-a333-48cd-8563-8d64b1ec333d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:49.058778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868402fd-a333-48cd-8563-8d64b1ec333d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:49.101809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:49.111811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:49.127232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da4857f-a65c-450d-81e8-134f62037445 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:49.134122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:49.136139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.136666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.144868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.179144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:49.182578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:49.203817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:49.204931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:49.205851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:49.205851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:49.205851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:49.208439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 258/504] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:49.216170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:49.231322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:49.232016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.232302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.233114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.234042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:49.234358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:49.235931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:49.276051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 247/505] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Oct 19 00:27:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 469 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:49.287640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:49.400307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201eb99c-c4db-43fd-8f5c-a621c3217d24 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:49.447330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cfd5a3-a60e-44b9-a1a8-a508c4b9ea74 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:49.454859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430f990b-a7d1-4079-a70e-ff0be8dc1f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:49.455485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f990b-a7d1-4079-a70e-ff0be8dc1f9b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.455762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f990b-a7d1-4079-a70e-ff0be8dc1f9b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.456038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f990b-a7d1-4079-a70e-ff0be8dc1f9b None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.456701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430f990b-a7d1-4079-a70e-ff0be8dc1f9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:49.457002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430f990b-a7d1-4079-a70e-ff0be8dc1f9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:49.458420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430f990b-a7d1-4079-a70e-ff0be8dc1f9b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:49.478524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 259/506] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:49.478524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0340fa9-d816-4d53-8280-95d2c4f3cebe None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.478524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0340fa9-d816-4d53-8280-95d2c4f3cebe None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.478524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0340fa9-d816-4d53-8280-95d2c4f3cebe None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.478524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 260/507] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:49.479718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b15e5f-8958-41df-9705-5529909fee3a None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.479849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b15e5f-8958-41df-9705-5529909fee3a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.480630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b15e5f-8958-41df-9705-5529909fee3a None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.481970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 261/508] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:49.485940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b15e5f-8958-41df-9705-5529909fee3a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:49.516140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6631dd35-c56a-4804-af40-f03dba94aecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:49.517201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6631dd35-c56a-4804-af40-f03dba94aecd None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.517617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6631dd35-c56a-4804-af40-f03dba94aecd None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.518020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6631dd35-c56a-4804-af40-f03dba94aecd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.533275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6631dd35-c56a-4804-af40-f03dba94aecd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:49.535296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6631dd35-c56a-4804-af40-f03dba94aecd None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:49.550540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-6631dd35-c56a-4804-af40-f03dba94aecd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:49.556766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:49.559051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:49.562424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:49.562424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:49.562424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:49.565958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 248/509] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:49.568191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ebee9-ae2e-48a3-add6-4fcf119960c2 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.568633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ebee9-ae2e-48a3-add6-4fcf119960c2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.569072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ebee9-ae2e-48a3-add6-4fcf119960c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.594415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-b47ebee9-ae2e-48a3-add6-4fcf119960c2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:49.596459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b47ebee9-ae2e-48a3-add6-4fcf119960c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:49.601023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:49.682399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b15e5f-8958-41df-9705-5529909fee3a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:49.689007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 249/510] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:49.690621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c7975-ac05-41d2-9ba2-681c20403402 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.691054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c7975-ac05-41d2-9ba2-681c20403402 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.691453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c7975-ac05-41d2-9ba2-681c20403402 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.725029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-079c7975-ac05-41d2-9ba2-681c20403402 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:49.728289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-079c7975-ac05-41d2-9ba2-681c20403402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:49.751788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:49.752404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:49.752873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:49.753392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:49.766408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:49.769079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:49.791127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:49.792859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:49.794603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 262/511] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:49.800725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:49.865111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 250/512] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:49.865111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c7975-ac05-41d2-9ba2-681c20403402 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:50.028804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:50.036538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:50.069331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71cd3e7-8e6c-411e-a4a6-dbdbf7b640b0 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:50.094799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c7975-ac05-41d2-9ba2-681c20403402 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:50.130299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:50.131165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.131446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.131721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.132794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:50.133099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:50.134813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:50.141872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:50.142849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.143222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.143565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.144342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:50.144725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:50.146638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:50.149673 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 263/513] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:50.155505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:50.164167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 251/514] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:50.165121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:50.315817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6157198-3402-482a-b528-9a909d0e4806 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:50.323605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77204b23-1ca7-46f6-bc35-46a1e315d5c2 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:50.358128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:50.368014 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:50.368686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.368970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.369252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.374156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:50.374881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.375166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.375443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] PATH_INFO: `/v3/services/network` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.376055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:50.377540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:50.377999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:50.379676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:50.390644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:50.392739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:50.395530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 264/515] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:50 2022] GET /identity/v3/services/network => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:50.398688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:50.398912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:50.401394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 252/516] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:50.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:50.433701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:50.527178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594cf529-1ee3-4513-b2ff-f1d85546fbde None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:50.544845 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc8dd38-f729-4a17-96c3-997f2c2a7e2c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:50.585735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6809862a-c3c8-410e-b362-d5842c41169f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:50.586431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6809862a-c3c8-410e-b362-d5842c41169f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.586818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6809862a-c3c8-410e-b362-d5842c41169f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.587230 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6809862a-c3c8-410e-b362-d5842c41169f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.588080 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6809862a-c3c8-410e-b362-d5842c41169f None admin] RBAC: Adding query filter params (role.id=8b6c794a10cf4fcdb33edf34796d4819, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=33b9c777f8284edeb758b00de044fd45) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:50.588517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6809862a-c3c8-410e-b362-d5842c41169f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:50.590256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6809862a-c3c8-410e-b362-d5842c41169f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:50.591124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:50.592214 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.592712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.593156 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.594137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:50.596619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:50.598236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 265/517] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:27:50 2022] GET /identity/v3/role_assignments?user.id=33b9c777f8284edeb758b00de044fd45&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=8b6c794a10cf4fcdb33edf34796d4819 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:50.599581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c479a0-b4d4-414e-aaf1-e2c41eec617e None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.599871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c479a0-b4d4-414e-aaf1-e2c41eec617e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.600148 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c479a0-b4d4-414e-aaf1-e2c41eec617e None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.601696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 266/518] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:50.602357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c479a0-b4d4-414e-aaf1-e2c41eec617e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:50.621395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:50.623732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:27:50.625961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:50.627921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 253/519] 10.222.0.26 () {56 vars in 1129 bytes} [Wed Oct 19 00:27:50 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:50.627921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:50.719093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:50.726338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:50.747700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c479a0-b4d4-414e-aaf1-e2c41eec617e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:50.799535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cbff34-b0aa-4c33-8915-991b332e8afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:50.800177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbff34-b0aa-4c33-8915-991b332e8afe None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.800463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbff34-b0aa-4c33-8915-991b332e8afe None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.800729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbff34-b0aa-4c33-8915-991b332e8afe None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.801509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cbff34-b0aa-4c33-8915-991b332e8afe None admin] RBAC: Adding query filter params (name=network) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:50.801814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cbff34-b0aa-4c33-8915-991b332e8afe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:50.803300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cbff34-b0aa-4c33-8915-991b332e8afe None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:50.814534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 267/520] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:27:50 2022] GET /identity/v3/services?name=network => generated 124 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:50.814534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13b82d-7d9f-4a89-b7e7-12caf8d4c437 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.814534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13b82d-7d9f-4a89-b7e7-12caf8d4c437 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.814916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649d0f02-02d8-46f2-8858-49e1d6a13c90 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:50.815005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13b82d-7d9f-4a89-b7e7-12caf8d4c437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.832296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1c13b82d-7d9f-4a89-b7e7-12caf8d4c437 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:50.833617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1c13b82d-7d9f-4a89-b7e7-12caf8d4c437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:50.875873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:50.876783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:50.877249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:50.877700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:50.878942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:50.879458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:50.882022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:50.909743 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 254/521] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:50.909743 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:50.934649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 268/522] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:50.934649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c13b82d-7d9f-4a89-b7e7-12caf8d4c437 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:51.067455 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:51.083342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae9c97-66c7-4d8e-a45f-c07cf8f1739e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:51.106757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:51.126765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c13b82d-7d9f-4a89-b7e7-12caf8d4c437 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:51.151761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:51.152708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:51.153185 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:51.153657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:51.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:51.155352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:51.158279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:51.167576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 255/523] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Oct 19 00:27:50 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:51.169074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:51.199866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:51.201210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:51.202354 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:51.203099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:51.204354 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] RBAC: Adding query filter params (type=network) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:51.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:51.208646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:51.221823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 269/524] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:27:50 2022] GET /identity/v3/services?type=network => generated 124 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:51.232167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:51.375794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c219f0-8eee-46d6-8786-7d6136414430 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:51.378146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae543a65-c392-4da0-a853-869aac4f9fee None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:51.424062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:51.425306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:51.425855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:51.426333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:51.427771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:51.428329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:51.430499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:51.442871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 270/525] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Oct 19 00:27:51 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:51.442871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:51.455738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:51.463722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5694c11e-1707-4e15-845f-cf8cd657ad9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:51.470238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5694c11e-1707-4e15-845f-cf8cd657ad9d None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:51.470857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5694c11e-1707-4e15-845f-cf8cd657ad9d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:51.471399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5694c11e-1707-4e15-845f-cf8cd657ad9d None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/41009aec3b1843aa80323c374cc01461/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:51.486093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5694c11e-1707-4e15-845f-cf8cd657ad9d None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=41009aec3b1843aa80323c374cc01461, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:51.489708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5694c11e-1707-4e15-845f-cf8cd657ad9d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:51.527864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 256/526] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:51 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/41009aec3b1843aa80323c374cc01461/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:51.724422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868aa9e-a510-43bc-9292-f1f1dcb22ad7 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:51.794352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:51.795594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:51.796299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:51.796770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:51.811266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:51.813180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:51.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:51.831067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:51.832131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:51.832131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:51.832131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:51.836635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 271/527] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 395 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:51.844541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5694c11e-1707-4e15-845f-cf8cd657ad9d None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:51.888988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:51.979399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5694c11e-1707-4e15-845f-cf8cd657ad9d None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:52.034914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6a6ee5-9c1a-4c81-86b1-419d0591013c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:52.035966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a6ee5-9c1a-4c81-86b1-419d0591013c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:52.036434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a6ee5-9c1a-4c81-86b1-419d0591013c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:52.036896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a6ee5-9c1a-4c81-86b1-419d0591013c None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:52.037752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a6ee5-9c1a-4c81-86b1-419d0591013c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:52.038252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a6ee5-9c1a-4c81-86b1-419d0591013c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:52.040039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a6ee5-9c1a-4c81-86b1-419d0591013c None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:52.060835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 257/528] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:52.070265 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:52.236075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:52.351780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ecb8f-6b15-412e-b9fc-1e3ee9a566c0 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:52.444326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894541c3-4942-4024-b28b-3470b6b6a29c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:52.445551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894541c3-4942-4024-b28b-3470b6b6a29c None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:52.446138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894541c3-4942-4024-b28b-3470b6b6a29c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:52.446762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894541c3-4942-4024-b28b-3470b6b6a29c None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:52.448144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894541c3-4942-4024-b28b-3470b6b6a29c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:52.452060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894541c3-4942-4024-b28b-3470b6b6a29c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:52.485781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 272/529] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:27:52 2022] POST /identity/v3/users => generated 254 bytes in 417 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:27:53.177673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bec383-c38d-4f42-90b5-7ca6c98c91a7 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:53.178767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bec383-c38d-4f42-90b5-7ca6c98c91a7 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:53.179272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bec383-c38d-4f42-90b5-7ca6c98c91a7 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:53.181140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 258/530] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:53.187702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4b122e-f62a-4427-906c-7e71987342e8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:53.188656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4b122e-f62a-4427-906c-7e71987342e8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:53.189141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4b122e-f62a-4427-906c-7e71987342e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:53.210060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8b4b122e-f62a-4427-906c-7e71987342e8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:53.211676 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8b4b122e-f62a-4427-906c-7e71987342e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:53.298697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 273/531] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:53.583917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bec383-c38d-4f42-90b5-7ca6c98c91a7 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:53.726467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:53.736011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bec383-c38d-4f42-90b5-7ca6c98c91a7 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:53.777866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b204ba2-dd54-4861-8e15-2c9b26797ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:53.778524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b204ba2-dd54-4861-8e15-2c9b26797ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:53.778807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b204ba2-dd54-4861-8e15-2c9b26797ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:53.779081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b204ba2-dd54-4861-8e15-2c9b26797ec7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:53.791497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b204ba2-dd54-4861-8e15-2c9b26797ec7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-6b204ba2-dd54-4861-8e15-2c9b26797ec7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:53.796525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:53.797592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:53.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:53.802613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 259/532] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:27:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:53.804329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4b122e-f62a-4427-906c-7e71987342e8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:53.860585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:53.941864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4b122e-f62a-4427-906c-7e71987342e8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:53.992750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:53.993374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:53.993651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:53.993920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:53.994614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:53.994919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:53.996443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:54.009377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 274/533] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:54.015561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b204ba2-dd54-4861-8e15-2c9b26797ec7 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:54.257348 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b204ba2-dd54-4861-8e15-2c9b26797ec7 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:54.338328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:54.339470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:54.339970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:54.340455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:54.369459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:54.372280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:54.403760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:54.406790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:54.408517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:54.408517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:54.408517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:54.408517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 260/534] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 393 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:54.412639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:54.604691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaddd349-e61b-4a20-b5ea-f6e63c0e381e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:54.606552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:54.671728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:54.672764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:54.673248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:54.673729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:54.675055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:54.675583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:54.678077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:54.712502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 275/535] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:54.718849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:54.750794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:54.898826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e70582-4d29-4a8f-b996-de818c65d101 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:54.944811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c8e819-1790-4360-9ce2-11263721eae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:54.945934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8e819-1790-4360-9ce2-11263721eae7 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:54.946451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8e819-1790-4360-9ce2-11263721eae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:54.947009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8e819-1790-4360-9ce2-11263721eae7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:54.963255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c8e819-1790-4360-9ce2-11263721eae7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:54.965315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c8e819-1790-4360-9ce2-11263721eae7 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-85c8e819-1790-4360-9ce2-11263721eae7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:54.984132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:54.985199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 00:27:54.986242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:54.989009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 261/536] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:27:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:54.995640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:55.081211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0037ad3-fa21-4f10-becd-95763543e2fe None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.081546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0037ad3-fa21-4f10-becd-95763543e2fe None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0037ad3-fa21-4f10-becd-95763543e2fe None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.084760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 262/537] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:27:55.089620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2cf73-739e-4d6d-8e49-5b69034da27c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.089959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2cf73-739e-4d6d-8e49-5b69034da27c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.090199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2cf73-739e-4d6d-8e49-5b69034da27c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.111921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-88d2cf73-739e-4d6d-8e49-5b69034da27c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:55.113271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-88d2cf73-739e-4d6d-8e49-5b69034da27c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:55.261754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:55.271059 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 263/538] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:55.277260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebe61b-b7e2-430c-ba05-c6b79f9beeda None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.277563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebe61b-b7e2-430c-ba05-c6b79f9beeda None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.277816 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebe61b-b7e2-430c-ba05-c6b79f9beeda None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.282671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 264/539] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:55 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:55.291245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4237978-f4e1-417c-9751-b68060cbf90f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.291730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4237978-f4e1-417c-9751-b68060cbf90f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.292218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4237978-f4e1-417c-9751-b68060cbf90f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.307839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6eb561-0d87-44a0-8303-628af0d32f83 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:55.349516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c4237978-f4e1-417c-9751-b68060cbf90f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:55.351345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c4237978-f4e1-417c-9751-b68060cbf90f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:55.384544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcd17a9-85e5-4b27-80df-2d8894d723e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:55.385511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd17a9-85e5-4b27-80df-2d8894d723e6 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.385955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd17a9-85e5-4b27-80df-2d8894d723e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.386402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd17a9-85e5-4b27-80df-2d8894d723e6 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.387709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcd17a9-85e5-4b27-80df-2d8894d723e6 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:55.388347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcd17a9-85e5-4b27-80df-2d8894d723e6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:55.391296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcd17a9-85e5-4b27-80df-2d8894d723e6 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:55.427268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 276/540] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Oct 19 00:27:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 433 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:55.429572 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ab2df-6220-4031-934b-f540d73507e5 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.430038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ab2df-6220-4031-934b-f540d73507e5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.430498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ab2df-6220-4031-934b-f540d73507e5 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.433320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 277/541] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:55.434713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ab2df-6220-4031-934b-f540d73507e5 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:55.480735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 265/542] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 00:27:55.483382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02716e2-350a-4548-bfcb-b6425f39c68c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.483612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:55.484355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02716e2-350a-4548-bfcb-b6425f39c68c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.484932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02716e2-350a-4548-bfcb-b6425f39c68c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c02716e2-350a-4548-bfcb-b6425f39c68c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:55.537940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c02716e2-350a-4548-bfcb-b6425f39c68c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:55.641850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ab2df-6220-4031-934b-f540d73507e5 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:55.679488 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 266/543] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 00:27:55.681539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02716e2-350a-4548-bfcb-b6425f39c68c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:55.684794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:55.685899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.686692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.687192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.700018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:55.702818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:55.723659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:55.724704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:55.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:55.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:55.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:55.733061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 278/544] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:55.735274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:55.821198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:55.876832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:55.904655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02716e2-350a-4548-bfcb-b6425f39c68c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:55.934190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c019aa8-7c8d-4981-8bc9-149466db456d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:55.976609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d4f52-63de-43a8-9896-6e25f2132a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:55.977255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d4f52-63de-43a8-9896-6e25f2132a94 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:55.977532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d4f52-63de-43a8-9896-6e25f2132a94 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:55.977804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d4f52-63de-43a8-9896-6e25f2132a94 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:55.978517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940d4f52-63de-43a8-9896-6e25f2132a94 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:55.979970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940d4f52-63de-43a8-9896-6e25f2132a94 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:56.018890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 267/545] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:27:55 2022] POST /identity/v3/services => generated 240 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:56.043262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:56.043901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:56.044249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:56.044518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:56.045314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:56.045665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:56.047204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:56.066150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 279/546] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:27:56.072244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d4f52-63de-43a8-9896-6e25f2132a94 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:56.073971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:56.287919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d4f52-63de-43a8-9896-6e25f2132a94 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:56.299752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceac4e99-b7d7-444e-a1ad-e6b254c857d2 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:56.312668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:56.335396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8cbd6-0e6c-4d9b-9e6e-2518c116958f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:56.335974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8cbd6-0e6c-4d9b-9e6e-2518c116958f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:56.336247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8cbd6-0e6c-4d9b-9e6e-2518c116958f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:56.336523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8cbd6-0e6c-4d9b-9e6e-2518c116958f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:56.345432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f8cbd6-0e6c-4d9b-9e6e-2518c116958f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:56.347074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f8cbd6-0e6c-4d9b-9e6e-2518c116958f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:56.348284 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:56.361221 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd7154-795b-4061-adfa-44c40a6c8bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-77f8cbd6-0e6c-4d9b-9e6e-2518c116958f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:56.361829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:56.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:56.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:56.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:56.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:56.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:56.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:56.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:56.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:56.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:56.364609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd7154-795b-4061-adfa-44c40a6c8bd2 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:56.364609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd7154-795b-4061-adfa-44c40a6c8bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:56.364609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd7154-795b-4061-adfa-44c40a6c8bd2 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:56.364792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 268/547] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:56.364792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962edb4e-abb9-4117-82a6-2ac5722ad694 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:56.364980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dd7154-795b-4061-adfa-44c40a6c8bd2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:56.365099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962edb4e-abb9-4117-82a6-2ac5722ad694 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:56.365099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962edb4e-abb9-4117-82a6-2ac5722ad694 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:56.366531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 269/548] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:56 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:56.367082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962edb4e-abb9-4117-82a6-2ac5722ad694 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:56.372003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dd7154-795b-4061-adfa-44c40a6c8bd2 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:56.399551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 280/549] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:27:56 2022] POST /identity/v3/services => generated 251 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:27:56.400743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd7154-795b-4061-adfa-44c40a6c8bd2 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:56.576857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962edb4e-abb9-4117-82a6-2ac5722ad694 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:56.578775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd7154-795b-4061-adfa-44c40a6c8bd2 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:56.629053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211138db-8bdd-45b3-add0-27cff97f186c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:56.629862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211138db-8bdd-45b3-add0-27cff97f186c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:56.629985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211138db-8bdd-45b3-add0-27cff97f186c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:56.630155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211138db-8bdd-45b3-add0-27cff97f186c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:56.646077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211138db-8bdd-45b3-add0-27cff97f186c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-211138db-8bdd-45b3-add0-27cff97f186c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:56.651948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:56.652947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:27:56.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:56.660853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:56.660853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:56.661035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 270/550] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:27:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:56.661107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:56.661107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:56.661107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:56.661107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:56.661354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda6e40c-9c36-4671-98e3-8bc02ec836d4 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:56.661354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda6e40c-9c36-4671-98e3-8bc02ec836d4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:56.661354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda6e40c-9c36-4671-98e3-8bc02ec836d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:56.661531 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:56.683876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 281/551] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:56.684917 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:56.686600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-cda6e40c-9c36-4671-98e3-8bc02ec836d4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:56.688054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-cda6e40c-9c36-4671-98e3-8bc02ec836d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:56.729406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:56.764465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:56.783473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 271/552] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:27:56.790520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda6e40c-9c36-4671-98e3-8bc02ec836d4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:56.838514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca2c9c5-e476-4b61-84bd-5518d5fbe56a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:56.874520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:56.874520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:56.874520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:56.874520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:56.875183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:56.875183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:56.876541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:56.896980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 282/553] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:27:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:56.907311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:57.014642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda6e40c-9c36-4671-98e3-8bc02ec836d4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:57.058229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c643a-010b-4333-b78c-dfceea09cbc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:57.059023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c643a-010b-4333-b78c-dfceea09cbc6 None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:57.059296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c643a-010b-4333-b78c-dfceea09cbc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:57.059574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c643a-010b-4333-b78c-dfceea09cbc6 None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/33b9c777f8284edeb758b00de044fd45/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:57.063689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c643a-010b-4333-b78c-dfceea09cbc6 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=33b9c777f8284edeb758b00de044fd45, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:57.065265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c643a-010b-4333-b78c-dfceea09cbc6 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:57.085159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 272/554] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:27:56 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/33b9c777f8284edeb758b00de044fd45/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:27:57.121269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e86a724-22de-4e09-9e27-11db9deeac28 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:57.131326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:57.165011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:57.165551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:57.165828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:57.166095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:57.178450 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:57.179963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:57.196887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:57.198717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:57.200366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:57.200366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:57.200366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:57.200366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 283/555] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:57.210444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c643a-010b-4333-b78c-dfceea09cbc6 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:57.258005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:57.305336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:57.367691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c643a-010b-4333-b78c-dfceea09cbc6 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:57.453841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:57.455260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:57.455564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:57.455837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:57.456477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:57.456781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:57.458196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:57.462942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf6f39-5dda-4c31-8f73-abd89ace181a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:57.475413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 273/556] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:57.484895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:57.532201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38609b0d-9462-4368-8cfd-eb4420034658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:57.533183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38609b0d-9462-4368-8cfd-eb4420034658 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:57.533666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38609b0d-9462-4368-8cfd-eb4420034658 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:57.534117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38609b0d-9462-4368-8cfd-eb4420034658 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:57.566523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38609b0d-9462-4368-8cfd-eb4420034658 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-38609b0d-9462-4368-8cfd-eb4420034658 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:27:57.577965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:57.579935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:57.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 284/557] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:27:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:57.586842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38609b0d-9462-4368-8cfd-eb4420034658 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:57.641926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:57.723534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:57.754079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9af66-84ee-4890-9e61-7e5bfccc2ca9 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:57.827850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38609b0d-9462-4368-8cfd-eb4420034658 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:57.834642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:57.835307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:57.835588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:57.835862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:57.855774 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:57.857691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:57.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:57.881689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:27:57.882680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:57.883881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 274/558] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Oct 19 00:27:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 400 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:57.889847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:57.896976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:57.898016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:57.898528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:57.899015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:57.900063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] RBAC: Adding query filter params (name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:57.900697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:57.907216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:57.931392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 285/559] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:27:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:57.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:58.123883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2247be10-a733-4007-99be-476bbea652b2 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:58.146275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:58.186572 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b8852-e9be-49a4-be45-39e56c8cee4e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:58.200262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:58.200886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:58.201919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:58.202515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:58.202799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:58.203100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:58.204572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:58.241772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 275/560] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Oct 19 00:27:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:58.247452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:58.251319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:58.281268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:58.281934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:58.282208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:58.282477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:58.311828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:58.313283 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:58.350910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:58.351934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:58.352822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:58.352822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:58.352822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:58.353579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 286/561] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 416 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:58.358277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:58.470255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f592805-3ffd-4e65-b596-589d21814ecc None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:58.571583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d600bc28-aaa2-41df-99bc-2165c3a5c567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:58.572281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600bc28-aaa2-41df-99bc-2165c3a5c567 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:58.572561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600bc28-aaa2-41df-99bc-2165c3a5c567 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:58.572832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600bc28-aaa2-41df-99bc-2165c3a5c567 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:58.619807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4286c24-d69c-4e4e-9c5a-9f79464b9bbb None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:58.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:58.713005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2240e46a-dec7-4d2b-a7b7-06a939391c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:58.713732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2240e46a-dec7-4d2b-a7b7-06a939391c39 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:58.714015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2240e46a-dec7-4d2b-a7b7-06a939391c39 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:58.714285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2240e46a-dec7-4d2b-a7b7-06a939391c39 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:58.715001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2240e46a-dec7-4d2b-a7b7-06a939391c39 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:58.717699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2240e46a-dec7-4d2b-a7b7-06a939391c39 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:58.719244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2240e46a-dec7-4d2b-a7b7-06a939391c39 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:58.735821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 287/562] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 378 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:58.737643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1d919d-31ef-49d0-840a-ccbdab797242 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:58.738187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1d919d-31ef-49d0-840a-ccbdab797242 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:58.738742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1d919d-31ef-49d0-840a-ccbdab797242 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:58.744785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d600bc28-aaa2-41df-99bc-2165c3a5c567 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:58.745151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 288/563] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:58.746310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d600bc28-aaa2-41df-99bc-2165c3a5c567 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:58.746539 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1d919d-31ef-49d0-840a-ccbdab797242 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:58.748647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:58.883573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1d919d-31ef-49d0-840a-ccbdab797242 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:58.886126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 276/564] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:27:58 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 639 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 19 00:27:58.887805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a725b4-566c-41de-95e2-59c3adb8d49a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:58.888336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a725b4-566c-41de-95e2-59c3adb8d49a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:58.890003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a725b4-566c-41de-95e2-59c3adb8d49a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:58.910686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-71a725b4-566c-41de-95e2-59c3adb8d49a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:58.912254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-71a725b4-566c-41de-95e2-59c3adb8d49a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:58.933043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:58.934023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:58.934305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:58.934616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:58.947163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:58.949362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:27:58.966839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:58.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 289/565] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Oct 19 00:27:58 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:58.970195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.014594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 277/566] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:58 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:27:59.014594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a725b4-566c-41de-95e2-59c3adb8d49a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.044758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:59.058971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:59.123308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a725b4-566c-41de-95e2-59c3adb8d49a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.132383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cef4332-b650-4c0d-918a-a0b93ef15fa8 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.160122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348f00af-7f18-4714-aa40-8d41128329df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.160736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f00af-7f18-4714-aa40-8d41128329df None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.161011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f00af-7f18-4714-aa40-8d41128329df None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.161278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f00af-7f18-4714-aa40-8d41128329df None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.162017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348f00af-7f18-4714-aa40-8d41128329df None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:59.162316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348f00af-7f18-4714-aa40-8d41128329df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.163895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348f00af-7f18-4714-aa40-8d41128329df None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.175044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef47c65-684a-400b-95a5-4a7036dd9d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.175792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef47c65-684a-400b-95a5-4a7036dd9d39 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.176073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef47c65-684a-400b-95a5-4a7036dd9d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.176355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef47c65-684a-400b-95a5-4a7036dd9d39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.184962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 278/567] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:59.186583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f43fb-4a02-40c1-968e-3adec52e9dbb None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.186860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f43fb-4a02-40c1-968e-3adec52e9dbb None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.187150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f43fb-4a02-40c1-968e-3adec52e9dbb None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.188652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 279/568] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:27:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:27:59.190140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90f43fb-4a02-40c1-968e-3adec52e9dbb None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.195320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef47c65-684a-400b-95a5-4a7036dd9d39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.197775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef47c65-684a-400b-95a5-4a7036dd9d39 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-3ef47c65-684a-400b-95a5-4a7036dd9d39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:59.223483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:59.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:59.225914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:59.225914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:59.225914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:27:59.225914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 290/569] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:59.230517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afee236-07ca-40f8-8c63-f36d29e3007d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.230517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afee236-07ca-40f8-8c63-f36d29e3007d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.230517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afee236-07ca-40f8-8c63-f36d29e3007d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.250575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9afee236-07ca-40f8-8c63-f36d29e3007d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:27:59.251999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9afee236-07ca-40f8-8c63-f36d29e3007d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:27:59.318416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:59.332327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90f43fb-4a02-40c1-968e-3adec52e9dbb None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.358660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 291/570] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:27:59 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:27:59.359921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afee236-07ca-40f8-8c63-f36d29e3007d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.373737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.374730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.375003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.375373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.388055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.390323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.402409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:27:59.410325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:59.411438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:27:59.417289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:27:59.417289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:27:59.417289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:59.417289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 280/571] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:27:59.417289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.505253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afee236-07ca-40f8-8c63-f36d29e3007d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.520303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ddde33-cad9-4786-b814-f3b37e7d4ded None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.558080 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.558973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.559630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.561450 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.562765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:59.563566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.565990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.569317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.569909 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.570247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.570638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] PATH_INFO: `/v3/services/network` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.571300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.572739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.582088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 292/572] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:59.583235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:59.587181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:59.588254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:59.589168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 281/573] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/services/network => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:59.589989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.600778 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:59.697036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:59.706673 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2bdd4b-8143-42ab-b281-d8dea97296be None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.707209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4066781-0b73-4691-8c9b-3cdb6a314482 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.742616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.743220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.743531 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.743763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.744442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:59.744765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.745893 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.746097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.746531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.746811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.747079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.754927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.756363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.759051 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 293/574] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:59.759869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:27:59.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:59.774229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:59.775302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:27:59.775302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:27:59.775302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:27:59.775302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:27:59.775302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:27:59.775302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:27:59.775302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:27:59.775302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:59.778647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 282/575] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:59.780526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.869324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa70d46f-66dc-440e-89ba-8121cb89a489 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.869548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:59.906175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.906820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.907114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.907371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.907995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] RBAC: Adding query filter params (name=network) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:27:59.908297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.909678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.914359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 294/576] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/services?name=network => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:27:59.915184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:27:59.916532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d2b5ef-c325-4689-9a9e-87ffc256cb35 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:27:59.934786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:27:59.959090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:27:59.959742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:27:59.959998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:27:59.960267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:27:59.960848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:27:59.962218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:27:59.972004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:27:59.973407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:27:59.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 283/577] 10.222.0.26 () {56 vars in 1129 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:27:59.975232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.034784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf1c477-d1c1-4af8-8759-0594dc4092b5 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.074300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.075100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.075383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.075658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.083532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.084697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d1edcc-3bc9-4264-ae12-8dc48f82e49f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.085000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:28:00.097095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:00.098551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 295/578] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:00.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.122468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:00.130605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.131142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.131483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.131763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.132463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:00.132769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.134217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.152265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 284/579] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:27:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:00.152265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.198818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:00.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a49b3d9-d77e-4a20-9e0e-a3ac4e26ad8b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.256696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0b3d7-b688-4276-b140-93496abb1199 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.264740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.265482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.265661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.265900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.266553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] RBAC: Adding query filter params (type=network) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:00.266858 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.268232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.274519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 296/580] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:00 2022] GET /identity/v3/services?type=network => generated 350 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:00.275235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.299350 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.299965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.300249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.300525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.301151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:00.301459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.302860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.307989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 285/581] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Oct 19 00:28:00 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:00.312237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.374448 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:00.389692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b09d614-eb07-491d-afc1-32564e550f33 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.425072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90bf88e-940a-42f0-8117-b5024dd0d160 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.426951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.427666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.427964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.428349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.429047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:00.429351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.430797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.449578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 297/582] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:28:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:00.449578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.449578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:00.471137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.471846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.472189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.472730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.473513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=41009aec3b1843aa80323c374cc01461) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:00.473817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.475278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.490522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 286/583] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:28:00 2022] GET /identity/v3/role_assignments?user.id=41009aec3b1843aa80323c374cc01461&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:00.490522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.592026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e449ad60-8900-4c41-8b77-b242a577b525 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.635356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2257c26-27b1-4bea-943f-999ba08654f6 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.635874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.636388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.636744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.637216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.638140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] RBAC: Adding query filter params (interface=public, service_id=0fda280bce69483089e8fa3a711de805, region_id=RegionOne) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:00.638776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.640354 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.654686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 298/584] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Oct 19 00:28:00 2022] GET /identity/v3/endpoints?service_id=0fda280bce69483089e8fa3a711de805&interface=public®ion_id=RegionOne => generated 194 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:00.655986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.656873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:00.677806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.679020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.679650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.680354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.681440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:00.682079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.683865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.696098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 287/585] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Oct 19 00:28:00 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:00.697215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.753529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:00.795170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ded53bf-a2b2-4168-8491-c54abdfb535e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.833999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4f871-a298-4833-bbc6-5e92f1ce42f3 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:00.834391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ccee25-da5c-4980-89be-f2214833e580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.835374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ccee25-da5c-4980-89be-f2214833e580 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.835906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ccee25-da5c-4980-89be-f2214833e580 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.836550 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ccee25-da5c-4980-89be-f2214833e580 None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/f7fcaf24d2ff423a90d9ea6747a67a05/roles/8b6c794a10cf4fcdb33edf34796d4819` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.854730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ccee25-da5c-4980-89be-f2214833e580 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=f7fcaf24d2ff423a90d9ea6747a67a05, role_id=8b6c794a10cf4fcdb33edf34796d4819)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.856323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ccee25-da5c-4980-89be-f2214833e580 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.878525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 299/586] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:28:00 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/f7fcaf24d2ff423a90d9ea6747a67a05/roles/8b6c794a10cf4fcdb33edf34796d4819 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:28:00.878735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1ecd8b-9fe6-4cff-adc9-e56273e9a18f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:00.878882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ccee25-da5c-4980-89be-f2214833e580 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:00.878972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1ecd8b-9fe6-4cff-adc9-e56273e9a18f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.878972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1ecd8b-9fe6-4cff-adc9-e56273e9a18f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.878972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1ecd8b-9fe6-4cff-adc9-e56273e9a18f None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.878972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1ecd8b-9fe6-4cff-adc9-e56273e9a18f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:00.886613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1ecd8b-9fe6-4cff-adc9-e56273e9a18f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:00.900910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 288/587] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:28:00 2022] GET /identity/v3/services => generated 772 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:00.902400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de06f3-4e6b-4d6e-91e5-ac1ba91d8092 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.902714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de06f3-4e6b-4d6e-91e5-ac1ba91d8092 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.902995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de06f3-4e6b-4d6e-91e5-ac1ba91d8092 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.905313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 289/588] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:00.912088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5357fe3d-77af-4b7b-a4d4-45a7547e2eab None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:00.912445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5357fe3d-77af-4b7b-a4d4-45a7547e2eab None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:00.912759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5357fe3d-77af-4b7b-a4d4-45a7547e2eab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:00.958999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5357fe3d-77af-4b7b-a4d4-45a7547e2eab None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:00.960383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5357fe3d-77af-4b7b-a4d4-45a7547e2eab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:01.068336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:01.119064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 290/589] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:00 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:01.120986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ccee25-da5c-4980-89be-f2214833e580 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:01.145920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:01.161983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:01.162644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:01.162952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:01.163231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:01.163951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] RBAC: Adding query filter params (interface=public, service_id=b850fc67a148422bb4b0e61dddeb1dbd, region_id=RegionOne) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:01.164260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:01.165653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:01.172412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 300/590] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Oct 19 00:28:00 2022] GET /identity/v3/endpoints?service_id=b850fc67a148422bb4b0e61dddeb1dbd&interface=public®ion_id=RegionOne => generated 194 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:01.177972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5357fe3d-77af-4b7b-a4d4-45a7547e2eab None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:01.298728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5357fe3d-77af-4b7b-a4d4-45a7547e2eab None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:01.342327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974aa515-40be-4e50-9dba-3028cf37c448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:01.343021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974aa515-40be-4e50-9dba-3028cf37c448 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:01.343305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974aa515-40be-4e50-9dba-3028cf37c448 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:01.343581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974aa515-40be-4e50-9dba-3028cf37c448 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:01.344210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974aa515-40be-4e50-9dba-3028cf37c448 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:01.345653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974aa515-40be-4e50-9dba-3028cf37c448 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:01.352611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 291/591] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:28:01 2022] GET /identity/v3/services => generated 772 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:01.594638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:01.702959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b17c0-2408-49b8-b50d-72aab92ac94b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:01.742033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a8707-f261-4ce9-829c-f1eca3ecad51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:01.742716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a8707-f261-4ce9-829c-f1eca3ecad51 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:01.742996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a8707-f261-4ce9-829c-f1eca3ecad51 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:01.743271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a8707-f261-4ce9-829c-f1eca3ecad51 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:01.757824 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98a8707-f261-4ce9-829c-f1eca3ecad51 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-d98a8707-f261-4ce9-829c-f1eca3ecad51 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:28:01.764756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:01.765731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:01.766757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 301/592] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:28:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:01.772109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974aa515-40be-4e50-9dba-3028cf37c448 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:01.841165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:01.924662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974aa515-40be-4e50-9dba-3028cf37c448 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:01.979530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:01.980156 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:01.980439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:01.980713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:01.981388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:01.981689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:01.983655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:02.000049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 292/593] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:28:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:02.006184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a8707-f261-4ce9-829c-f1eca3ecad51 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:02.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:02.203896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a8707-f261-4ce9-829c-f1eca3ecad51 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:02.261031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:02.261688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:02.261963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:02.262242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:02.275076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:02.276630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:02.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:02.296231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:02.297786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:02.297786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:02.297786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:02.297786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 302/594] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:02.301607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:02.419275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5e3d71-c30f-44bd-8044-99a1539eaab6 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:02.430888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:02.460964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:02.461643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:02.461929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:02.462241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:02.462981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:02.463340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:02.464886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:02.479680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 293/595] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:02.487764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:02.644475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:02.659067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c9389b-c053-4cbf-8dec-eaee5aa2d106 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:02.714358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:02.715133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:02.715485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:02.715763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:02.728551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:02.730090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:28:02.745703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:02.746769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:02.748300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 303/596] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:28:02 2022] GET /identity/v3/users/swift => generated 83 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:02.754920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:02.892859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ff3db-0ec0-45ed-be8b-18e65774c2d2 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:02.988461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58766e6-a62d-4975-aeff-0911ec7d560f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:02.989412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58766e6-a62d-4975-aeff-0911ec7d560f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:02.989881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58766e6-a62d-4975-aeff-0911ec7d560f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:02.990349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58766e6-a62d-4975-aeff-0911ec7d560f None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:02.991700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58766e6-a62d-4975-aeff-0911ec7d560f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:02.992255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58766e6-a62d-4975-aeff-0911ec7d560f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:02.994957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58766e6-a62d-4975-aeff-0911ec7d560f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:03.041320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 294/597] 10.222.0.26 () {56 vars in 1135 bytes} [Wed Oct 19 00:28:02 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:03.048760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:03.285163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19501704-2bc0-4fa1-9296-23f2e421b10b None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:03.286169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19501704-2bc0-4fa1-9296-23f2e421b10b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:03.286973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19501704-2bc0-4fa1-9296-23f2e421b10b None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:03.289941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 295/598] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:03.295810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3f25ee-eefd-46e1-86c7-b30a764ee62b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:03.296422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3f25ee-eefd-46e1-86c7-b30a764ee62b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:03.296916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3f25ee-eefd-46e1-86c7-b30a764ee62b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:03.297340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7548effe-902e-48ec-be26-600e25e0095b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:03.317411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6a3f25ee-eefd-46e1-86c7-b30a764ee62b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:03.319027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6a3f25ee-eefd-46e1-86c7-b30a764ee62b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:03.347548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:03.377678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d326bdf8-7a75-4eeb-a4b7-bb7d6785138b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:03.378312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d326bdf8-7a75-4eeb-a4b7-bb7d6785138b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:03.378620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d326bdf8-7a75-4eeb-a4b7-bb7d6785138b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:03.378903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d326bdf8-7a75-4eeb-a4b7-bb7d6785138b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:03.445376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:03.480414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 296/599] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:03.608426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d326bdf8-7a75-4eeb-a4b7-bb7d6785138b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:03.611058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d326bdf8-7a75-4eeb-a4b7-bb7d6785138b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:03.800914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 304/600] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:28:03 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 753 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 00:28:03.806866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3f25ee-eefd-46e1-86c7-b30a764ee62b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:03.871815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d326bdf8-7a75-4eeb-a4b7-bb7d6785138b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:03.942467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3f25ee-eefd-46e1-86c7-b30a764ee62b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:03.993432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb953516-ecb2-422e-9840-92faefd7c3eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:03.994434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb953516-ecb2-422e-9840-92faefd7c3eb None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:03.995000 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb953516-ecb2-422e-9840-92faefd7c3eb None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:03.995503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb953516-ecb2-422e-9840-92faefd7c3eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.007950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:04.008935 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb953516-ecb2-422e-9840-92faefd7c3eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:04.011188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb953516-ecb2-422e-9840-92faefd7c3eb None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-cb953516-ecb2-422e-9840-92faefd7c3eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:04.029782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:04.030981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:04.032045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:04.032045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:04.032045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:04.034464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 297/601] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:04.036015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3085a853-0534-4b9a-91b4-023263cea666 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.036485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3085a853-0534-4b9a-91b4-023263cea666 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.036959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3085a853-0534-4b9a-91b4-023263cea666 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.038402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 298/602] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:04.043740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3085a853-0534-4b9a-91b4-023263cea666 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:04.046888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:04.096081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d326bdf8-7a75-4eeb-a4b7-bb7d6785138b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:04.165416 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43644bb9-3605-4718-9870-0c5c3c2a49e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:04.167950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43644bb9-3605-4718-9870-0c5c3c2a49e2 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.168870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43644bb9-3605-4718-9870-0c5c3c2a49e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.170226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43644bb9-3605-4718-9870-0c5c3c2a49e2 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.171609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43644bb9-3605-4718-9870-0c5c3c2a49e2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:04.174695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43644bb9-3605-4718-9870-0c5c3c2a49e2 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:04.175783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3085a853-0534-4b9a-91b4-023263cea666 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-43644bb9-3605-4718-9870-0c5c3c2a49e2 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:04.199234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:04.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:04.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:04.204623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 305/603] 10.222.0.26 () {56 vars in 1132 bytes} [Wed Oct 19 00:28:03 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:04.205861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71171fd0-ecb3-4e20-9a6d-4e1b7828d9d7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.205861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71171fd0-ecb3-4e20-9a6d-4e1b7828d9d7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.205861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71171fd0-ecb3-4e20-9a6d-4e1b7828d9d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.231010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:04.231623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.231896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.232861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.233665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:04.234119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:04.236442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:04.244651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-71171fd0-ecb3-4e20-9a6d-4e1b7828d9d7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:04.248970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-71171fd0-ecb3-4e20-9a6d-4e1b7828d9d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:04.252039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 299/604] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:04.253465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:04.328943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:04.378511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 306/605] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:04 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:04.378511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71171fd0-ecb3-4e20-9a6d-4e1b7828d9d7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:04.380585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093b695-2f76-41d1-beb6-e653efdb5eef None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:04.436530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f05e85-680e-41a0-9a25-88a98a2f3719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:04.437410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05e85-680e-41a0-9a25-88a98a2f3719 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.437701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05e85-680e-41a0-9a25-88a98a2f3719 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.437980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f05e85-680e-41a0-9a25-88a98a2f3719 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.438761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f05e85-680e-41a0-9a25-88a98a2f3719 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:04.439154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f05e85-680e-41a0-9a25-88a98a2f3719 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:04.441096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f05e85-680e-41a0-9a25-88a98a2f3719 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:04.442993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:04.449921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 300/606] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:04 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:04.452553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec2dd4d-a448-4fcd-8b70-fdb110ecec21 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.452966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec2dd4d-a448-4fcd-8b70-fdb110ecec21 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.453300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec2dd4d-a448-4fcd-8b70-fdb110ecec21 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.455217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 301/607] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:04.461247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a4481-2c79-4ed9-a4c3-8ed3a586235b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.461529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a4481-2c79-4ed9-a4c3-8ed3a586235b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.461826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a4481-2c79-4ed9-a4c3-8ed3a586235b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.486874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-897a4481-2c79-4ed9-a4c3-8ed3a586235b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:04.488355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-897a4481-2c79-4ed9-a4c3-8ed3a586235b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:04.546309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71171fd0-ecb3-4e20-9a6d-4e1b7828d9d7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:04.590149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76825a5-f69d-426d-8e67-51ea0821ab88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:04.590842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76825a5-f69d-426d-8e67-51ea0821ab88 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.591167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76825a5-f69d-426d-8e67-51ea0821ab88 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.591987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76825a5-f69d-426d-8e67-51ea0821ab88 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.601768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76825a5-f69d-426d-8e67-51ea0821ab88 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:04.603537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76825a5-f69d-426d-8e67-51ea0821ab88 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:04.617419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 302/608] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:04 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-d76825a5-f69d-426d-8e67-51ea0821ab88 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:28:04.617884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:04.619010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:04.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:28:04.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:28:04.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:28:04.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:28:04.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:28:04.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:28:04.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:04.627804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:04.628230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897a4481-2c79-4ed9-a4c3-8ed3a586235b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:04.628390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 307/609] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:04.628390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edacb95e-128e-4554-8add-b3ef3a7152cd None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.628390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edacb95e-128e-4554-8add-b3ef3a7152cd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.628390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edacb95e-128e-4554-8add-b3ef3a7152cd None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.628390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 308/610] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:04.631972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edacb95e-128e-4554-8add-b3ef3a7152cd None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:04.693834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:04.747835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897a4481-2c79-4ed9-a4c3-8ed3a586235b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:04.792801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e41723d-70c2-4f0f-aae7-e7835eec8c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:04.794017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e41723d-70c2-4f0f-aae7-e7835eec8c44 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.795093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e41723d-70c2-4f0f-aae7-e7835eec8c44 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.795578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e41723d-70c2-4f0f-aae7-e7835eec8c44 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.796883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e41723d-70c2-4f0f-aae7-e7835eec8c44 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:04.797429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e41723d-70c2-4f0f-aae7-e7835eec8c44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:04.800174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e41723d-70c2-4f0f-aae7-e7835eec8c44 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:04.818522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 303/611] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:04 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:04.818522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3463c7b6-32d5-4a61-bed7-8a0217912c52 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.818522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3463c7b6-32d5-4a61-bed7-8a0217912c52 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.818522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3463c7b6-32d5-4a61-bed7-8a0217912c52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.820307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edacb95e-128e-4554-8add-b3ef3a7152cd None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:04.834952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:04.846805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-3463c7b6-32d5-4a61-bed7-8a0217912c52 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:04.849103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3463c7b6-32d5-4a61-bed7-8a0217912c52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:04.887923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:04.889760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:04.890290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:04.891129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:04.892547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:04.893109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:04.896180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:04.918534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 309/612] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:28:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:04.920587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:04.967190 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 304/613] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:04 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:04.968691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3463c7b6-32d5-4a61-bed7-8a0217912c52 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.070802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:05.120959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9167ebb-e68b-4900-9c18-288d0a64ee8d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.144711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3463c7b6-32d5-4a61-bed7-8a0217912c52 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.181397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.182465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.183396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.184352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] PATH_INFO: `/v3/services/network` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.185702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:05.187479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:05.196873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d877f0aa-0fa4-4933-919e-7672316ad3a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.197672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877f0aa-0fa4-4933-919e-7672316ad3a9 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.198047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877f0aa-0fa4-4933-919e-7672316ad3a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.198417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877f0aa-0fa4-4933-919e-7672316ad3a9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:05.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:05.201375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 00:28:05.202219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:05.205298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 310/614] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:04 2022] GET /identity/v3/services/network => generated 88 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:05.206327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.217850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d877f0aa-0fa4-4933-919e-7672316ad3a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-d877f0aa-0fa4-4933-919e-7672316ad3a9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:28:05.223946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:05.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:05.225846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 305/615] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:28:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:05.226571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d877f0aa-0fa4-4933-919e-7672316ad3a9 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.254107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:05.339997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1fb570-4963-48bb-93b4-53e33ed0eeb5 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.354027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d877f0aa-0fa4-4933-919e-7672316ad3a9 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.366770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:05.390788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.391772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.392235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.392686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.393627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=33b9c777f8284edeb758b00de044fd45) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:05.394123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:05.395781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:05.397972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.398821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.399099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.399371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.400006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] RBAC: Adding query filter params (name=network) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:05.400305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:05.401726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:05.407230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 306/616] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:05 2022] GET /identity/v3/services?name=network => generated 124 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:05.407981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.411854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 311/617] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:28:05 2022] GET /identity/v3/role_assignments?user.id=33b9c777f8284edeb758b00de044fd45&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 608 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:05.412592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.557440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491d27b-6a81-4f13-83ed-bb1611d972af None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.561279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:05.572950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8550376-79e3-4795-8b83-fee1d6ed9452 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.607820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.608625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.608996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.609274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.609950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] RBAC: Adding query filter params (name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:05.610241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:05.611690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:05.621998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 307/618] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:28:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:05.623304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.623706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.624448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.624735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.625008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.625608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:05.628058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:05.638160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:05.639913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:05.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 312/619] 10.222.0.26 () {56 vars in 1129 bytes} [Wed Oct 19 00:28:05 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:05.641718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.669795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:05.758937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f80baa-55e7-43d6-8da8-280a81de569a None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.800028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae38dca-caf3-47eb-a969-cee1511d71f1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.800920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.801577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.801985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.802269 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.803103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] RBAC: Adding query filter params (type=network) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:05.803418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:05.804998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:05.817801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 308/620] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:05 2022] GET /identity/v3/services?type=network => generated 350 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:05.819007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.842836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.843946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.844260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.844503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.857974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:05.859511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:05.883809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:05.885634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:05.887442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:05.887442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:05.887442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:05.887442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 313/621] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:05.887442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:05.902370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:05.937075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:05.945002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cc7b49-e449-4e4e-bf0c-f7107a5fdddd None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:05.984537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:05.985189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:05.985436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:05.985688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:05.986326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:05.986680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:05.988083 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:05.993305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 309/622] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Oct 19 00:28:05 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:05.994139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:06.028357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561553e5-f69f-4665-a217-fb51ec351103 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:06.098571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80005ad-8b2d-478d-8dc3-89b58ca4b7f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:06.100049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80005ad-8b2d-478d-8dc3-89b58ca4b7f7 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:06.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80005ad-8b2d-478d-8dc3-89b58ca4b7f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:06.101646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80005ad-8b2d-478d-8dc3-89b58ca4b7f7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:06.103693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80005ad-8b2d-478d-8dc3-89b58ca4b7f7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:06.108089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80005ad-8b2d-478d-8dc3-89b58ca4b7f7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:06.150955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 314/623] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:28:05 2022] POST /identity/v3/endpoints => generated 337 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:06.154860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80005ad-8b2d-478d-8dc3-89b58ca4b7f7 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:06.198513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1017855-827f-4640-b60a-61507e97db87 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:06.256910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:06.335251 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80005ad-8b2d-478d-8dc3-89b58ca4b7f7 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:06.339164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:06.339899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:06.340179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:06.340454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:06.343734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:06.346414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:06.348359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:06.380885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 310/624] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:06.387076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:06.419012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:06.427275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:06.428359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:06.428814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:06.429677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:06.431904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:06.432620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:06.436119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:06.443624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 315/625] 10.222.0.26 () {56 vars in 1124 bytes} [Wed Oct 19 00:28:06 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:06.451312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:06.530588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e5100-2fec-4542-a291-d246e8069620 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:06.627730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:06.628399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:06.628746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:06.629022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:06.656811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:06.659084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:06.661244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f9078c-cff8-45fe-a18c-d3c9e1dcde59 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:28:06.678533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:06.679593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 00:28:06.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:06.681130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 311/626] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Oct 19 00:28:06 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:06.687574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:06.712407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:06.735843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d810461f-7757-4ce4-a5b6-8e1911044181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:06.736921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d810461f-7757-4ce4-a5b6-8e1911044181 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:06.737418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d810461f-7757-4ce4-a5b6-8e1911044181 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:06.737920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d810461f-7757-4ce4-a5b6-8e1911044181 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:06.739221 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d810461f-7757-4ce4-a5b6-8e1911044181 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:06.741883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d810461f-7757-4ce4-a5b6-8e1911044181 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:06.782157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 316/627] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:28:06 2022] POST /identity/v3/endpoints => generated 346 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:28:06.850593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:06.879577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68521219-5d85-4bc9-965b-14acd2e8db54 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:06.943851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:06.944860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:06.945303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:06.945806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:06.947334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:06.947883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:06.950858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:06.976258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 312/628] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Oct 19 00:28:06 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:06.983532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d810461f-7757-4ce4-a5b6-8e1911044181 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:07.149931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d810461f-7757-4ce4-a5b6-8e1911044181 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:07.195928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1ae0b0-cbfa-4b78-b968-59f2055f4c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:07.196753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1ae0b0-cbfa-4b78-b968-59f2055f4c8c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:07.196753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1ae0b0-cbfa-4b78-b968-59f2055f4c8c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:07.196995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1ae0b0-cbfa-4b78-b968-59f2055f4c8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:07.263770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:07.287343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1ae0b0-cbfa-4b78-b968-59f2055f4c8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:07.289503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1ae0b0-cbfa-4b78-b968-59f2055f4c8c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:07.427132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 317/629] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:28:06 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 00:28:07.432996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:07.593062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25514a5-5231-42eb-98a0-0218b1493af5 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:07.593418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25514a5-5231-42eb-98a0-0218b1493af5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:07.593615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25514a5-5231-42eb-98a0-0218b1493af5 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:07.595343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 318/630] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:07.601492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7073a1a-1bf9-491a-be88-55e9a406aeaf None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:07.601984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7073a1a-1bf9-491a-be88-55e9a406aeaf None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:07.602283 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7073a1a-1bf9-491a-be88-55e9a406aeaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:07.635820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01a425e-1ffa-4013-ba42-5d0eeb91b223 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:07.644176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b7073a1a-1bf9-491a-be88-55e9a406aeaf None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:07.647309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b7073a1a-1bf9-491a-be88-55e9a406aeaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:07.684629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:07.685719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:07.686227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:07.686535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:07.690203 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:07.704439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:07.705958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:07.745429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:07.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:07.749270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:07.749270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:07.749270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:07.749270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 313/631] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:07.760517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:07.784297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 319/632] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:07 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:07.968409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e966de-ee68-42fa-8efa-4c0715aa9a06 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:08.009416 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f4c58-5ee4-4895-9bd5-a862c37c8760 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.009702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f4c58-5ee4-4895-9bd5-a862c37c8760 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.009988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f4c58-5ee4-4895-9bd5-a862c37c8760 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.011661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 320/633] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:08.017605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04466d1-1f2f-4510-8675-6121a42e7e9d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.017842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04466d1-1f2f-4510-8675-6121a42e7e9d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.018110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04466d1-1f2f-4510-8675-6121a42e7e9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.033606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:08.034220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.034517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.034818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.035450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:08.035750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:08.037144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:08.057736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 314/634] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:08.058006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f04466d1-1f2f-4510-8675-6121a42e7e9d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:08.058006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f04466d1-1f2f-4510-8675-6121a42e7e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:08.059568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:08.075034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:08.143725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:08.183378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 321/635] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:08 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:08.186166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04466d1-1f2f-4510-8675-6121a42e7e9d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:08.220519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0facc-8ad3-46e3-8931-5ad355a544f2 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:08.285025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:08.286748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.287532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.288305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.302504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:08.305775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:28:08.329693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:08.331425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:08.333137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:28:08.333137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:28:08.333137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:28:08.333137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:28:08.333137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:28:08.333137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:28:08.333137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:08.333137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:08.343030 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 315/636] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:08.344432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04466d1-1f2f-4510-8675-6121a42e7e9d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:08.348781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:08.384664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc61c5c6-6db6-4225-ba79-8a64eef1b0ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:08.385337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc61c5c6-6db6-4225-ba79-8a64eef1b0ef None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.385629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc61c5c6-6db6-4225-ba79-8a64eef1b0ef None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.385901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc61c5c6-6db6-4225-ba79-8a64eef1b0ef None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.386576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc61c5c6-6db6-4225-ba79-8a64eef1b0ef None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:08.387954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc61c5c6-6db6-4225-ba79-8a64eef1b0ef None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:08.405800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 322/637] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:28:08 2022] POST /identity/v3/services => generated 269 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:08.478189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc61c5c6-6db6-4225-ba79-8a64eef1b0ef None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:08.481461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:08.579913 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e298d1be-76f1-48de-9997-9330a088186b None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:08.597058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:08.612515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc61c5c6-6db6-4225-ba79-8a64eef1b0ef None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:08.646622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:08.647239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.647535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.648179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.648893 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:08.649475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:08.651141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:08.657528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:08.658314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.658763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.659171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.660471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:08.663155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:08.665865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 316/638] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:28:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:08.676675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:08.682185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:08.683227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:08.684092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:08.684832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 323/639] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:08 2022] GET /identity/v3/services/object-store => generated 93 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:08.695913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:08.795011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:08.831472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1178f343-35a0-40f5-9012-02143d04d2cc None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:08.859889 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee62817e-b9e0-4a7f-a1e5-be6d0d7d3030 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:08.874882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:08.875918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.876400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.876871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.877732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:08.878247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:08.880041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:08.891152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 324/640] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:08 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:08.894202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:08.932000 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a6b1cc-703d-4422-9774-72041563bee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:08.933280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6b1cc-703d-4422-9774-72041563bee5 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:08.933594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6b1cc-703d-4422-9774-72041563bee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:08.934088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6b1cc-703d-4422-9774-72041563bee5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:08.936273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a6b1cc-703d-4422-9774-72041563bee5 None admin] RBAC: Adding query filter params (role.id=8b6c794a10cf4fcdb33edf34796d4819, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=f7fcaf24d2ff423a90d9ea6747a67a05) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:08.936980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a6b1cc-703d-4422-9774-72041563bee5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:08.939842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a6b1cc-703d-4422-9774-72041563bee5 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:08.960287 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 317/641] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:28:08 2022] GET /identity/v3/role_assignments?user.id=f7fcaf24d2ff423a90d9ea6747a67a05&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=8b6c794a10cf4fcdb33edf34796d4819 => generated 608 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:09.020120 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80c67ae-ad59-4cc2-b504-1d37c7783a31 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:09.058049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:09.066104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08eb082-30cb-4f8b-9aa4-4ea0a94c194b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:09.067173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08eb082-30cb-4f8b-9aa4-4ea0a94c194b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:09.067744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08eb082-30cb-4f8b-9aa4-4ea0a94c194b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:09.068216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08eb082-30cb-4f8b-9aa4-4ea0a94c194b None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:09.069067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08eb082-30cb-4f8b-9aa4-4ea0a94c194b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:09.069570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08eb082-30cb-4f8b-9aa4-4ea0a94c194b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:09.071403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08eb082-30cb-4f8b-9aa4-4ea0a94c194b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:09.078012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 325/642] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:08 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:10.171556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97fa47-ac51-463e-a1fa-2934188fee12 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:10.172100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97fa47-ac51-463e-a1fa-2934188fee12 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:10.172514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97fa47-ac51-463e-a1fa-2934188fee12 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:10.175402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 318/643] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:10.184435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a51be-ae48-4db4-b4df-44f44b6b6561 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:10.185093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a51be-ae48-4db4-b4df-44f44b6b6561 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:10.185498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a51be-ae48-4db4-b4df-44f44b6b6561 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:10.217219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0d2a51be-ae48-4db4-b4df-44f44b6b6561 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:10.219227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0d2a51be-ae48-4db4-b4df-44f44b6b6561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:10.344609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 326/644] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:10 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:10.711219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97fa47-ac51-463e-a1fa-2934188fee12 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:10.766344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:10.853205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97fa47-ac51-463e-a1fa-2934188fee12 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:10.895950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:10.896983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:10.897464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:10.897930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:10.898759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:10.900352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:10.911675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:10.912781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:10.913664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:10.916656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 319/645] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:10 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:10.922527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2a51be-ae48-4db4-b4df-44f44b6b6561 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:11.056664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2a51be-ae48-4db4-b4df-44f44b6b6561 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:11.096698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:11.122521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01ca2ef-2a48-48bb-9679-e14fdc242a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:11.123431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ca2ef-2a48-48bb-9679-e14fdc242a4b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.123893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ca2ef-2a48-48bb-9679-e14fdc242a4b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.124362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ca2ef-2a48-48bb-9679-e14fdc242a4b None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.125541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01ca2ef-2a48-48bb-9679-e14fdc242a4b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:11.126085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01ca2ef-2a48-48bb-9679-e14fdc242a4b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:11.128358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01ca2ef-2a48-48bb-9679-e14fdc242a4b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:11.142649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 327/646] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:10 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:11.146803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:11.274774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f2889-a1d2-4fd4-b324-a4cb069528bc None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.275317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f2889-a1d2-4fd4-b324-a4cb069528bc None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.275675 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2f2889-a1d2-4fd4-b324-a4cb069528bc None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.278094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 328/647] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:11.288014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4714083-9ff7-47e5-b10c-3d1da323f27f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.288635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4714083-9ff7-47e5-b10c-3d1da323f27f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.289073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4714083-9ff7-47e5-b10c-3d1da323f27f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.318925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:11.319458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e4714083-9ff7-47e5-b10c-3d1da323f27f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:11.321020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e4714083-9ff7-47e5-b10c-3d1da323f27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:11.329500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea91b57d-61a4-41f9-af20-7a88f4527b30 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:11.368871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07a8d4-9fb9-4826-ae1c-7b0b7f14c1d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:11.369565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07a8d4-9fb9-4826-ae1c-7b0b7f14c1d3 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.369845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07a8d4-9fb9-4826-ae1c-7b0b7f14c1d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.370116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07a8d4-9fb9-4826-ae1c-7b0b7f14c1d3 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.370793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d07a8d4-9fb9-4826-ae1c-7b0b7f14c1d3 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:11.371091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d07a8d4-9fb9-4826-ae1c-7b0b7f14c1d3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:11.373128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d07a8d4-9fb9-4826-ae1c-7b0b7f14c1d3 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:11.378517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 320/648] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:11 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:11.382350 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f114a3ed-c593-4bb4-9e34-e98cd4e78d2e None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.382676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f114a3ed-c593-4bb4-9e34-e98cd4e78d2e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.382935 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f114a3ed-c593-4bb4-9e34-e98cd4e78d2e None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.384511 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 321/649] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:11.390908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b9288-84ad-4502-8a77-78b8e50f02fe None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.391252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b9288-84ad-4502-8a77-78b8e50f02fe None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.391506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b9288-84ad-4502-8a77-78b8e50f02fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.413609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-299b9288-84ad-4502-8a77-78b8e50f02fe None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:11.414904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-299b9288-84ad-4502-8a77-78b8e50f02fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:11.439806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 329/650] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:11 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:11.441269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5689e-f31e-4ff9-b36b-9cc9a99ff73f None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.441555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5689e-f31e-4ff9-b36b-9cc9a99ff73f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.441830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5689e-f31e-4ff9-b36b-9cc9a99ff73f None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.444321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 330/651] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:11.456912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9959a5-16ee-4343-87d4-0bbad272ba10 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.457359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9959a5-16ee-4343-87d4-0bbad272ba10 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.457761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9959a5-16ee-4343-87d4-0bbad272ba10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.480576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-db9959a5-16ee-4343-87d4-0bbad272ba10 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:11.482881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-db9959a5-16ee-4343-87d4-0bbad272ba10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:11.505378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 322/652] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:11 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:11.612369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 331/653] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:11 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:11.727495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299b9288-84ad-4502-8a77-78b8e50f02fe None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:11.789954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9959a5-16ee-4343-87d4-0bbad272ba10 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:11.837805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:11.855613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299b9288-84ad-4502-8a77-78b8e50f02fe None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:11.866957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:11.905568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:11.906210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:11.906522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:11.906769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:11.907360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:11.908744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:11.918135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:11.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:11.921908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 323/654] 10.222.0.26 () {56 vars in 1132 bytes} [Wed Oct 19 00:28:11 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:11.923019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:11.944195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9959a5-16ee-4343-87d4-0bbad272ba10 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:12.005228 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159025db-2b0f-406f-bd2f-41074efe59b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:12.006199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159025db-2b0f-406f-bd2f-41074efe59b4 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:12.006757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159025db-2b0f-406f-bd2f-41074efe59b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:12.007257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159025db-2b0f-406f-bd2f-41074efe59b4 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:12.008400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159025db-2b0f-406f-bd2f-41074efe59b4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:12.011144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159025db-2b0f-406f-bd2f-41074efe59b4 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:12.039515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 332/655] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:28:11 2022] POST /identity/v3/services => generated 241 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:28:12.041025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159025db-2b0f-406f-bd2f-41074efe59b4 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:12.120075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93afd1a0-d073-4d51-ae27-10e9fecd76f4 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:12.196385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc274ca-1c0f-45c9-88f5-b1e23868eecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:12.197116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc274ca-1c0f-45c9-88f5-b1e23868eecf None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:12.197406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc274ca-1c0f-45c9-88f5-b1e23868eecf None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:12.197840 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc274ca-1c0f-45c9-88f5-b1e23868eecf None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:12.217201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc274ca-1c0f-45c9-88f5-b1e23868eecf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-0cc274ca-1c0f-45c9-88f5-b1e23868eecf None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:28:12.227178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:12.229062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 00:28:12.231485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:12.235107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 324/656] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Oct 19 00:28:11 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:12.239694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc274ca-1c0f-45c9-88f5-b1e23868eecf None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:12.261050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:12.283237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159025db-2b0f-406f-bd2f-41074efe59b4 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:12.288507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:12.335106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:12.335739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:12.336021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:12.336285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:12.336935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:12.337242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:12.338695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:12.350631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 333/657] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:12 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:12.354392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:12.428148 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc274ca-1c0f-45c9-88f5-b1e23868eecf None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:12.507890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:12.509839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:12.511480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:12.512606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:12.516300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:12.517211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:12.521409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:12.534751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec3b8ba-9e4c-4268-b269-fdeca6b1c55a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:12.550675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 325/658] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Oct 19 00:28:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:12.558325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:12.602855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:12.604288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:12.605056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:12.605828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:12.607297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:12.608068 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:12.611036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:12.621106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 334/659] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:12 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:12.628213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:12.678661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:12.718250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:12.779254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc65e93-147e-4f90-9a29-c699f8198344 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:12.795866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68157f0a-2e92-4ac2-8d25-e3a69209c906 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:12.850126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:12.851551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:12.851727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:12.852168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:12.866697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:12.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:12.869129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:12.869887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:12.871662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] RBAC: Adding query filter params (interface=public, service_id=e36ed19f0143458aa49c7105fb86748b, region_id=RegionOne) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:12.873116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:12.874421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:12.876426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:12.877128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:12.888706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 335/660] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Oct 19 00:28:12 2022] GET /identity/v3/endpoints?service_id=e36ed19f0143458aa49c7105fb86748b&interface=public®ion_id=RegionOne => generated 194 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:12.896796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:12.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:12.908278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:12.909240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:12.909240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:12.909240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:12.916705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 326/661] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:12.918753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:13.005590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:13.013843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b042fdc-3fff-4575-a56e-3d1cb102cb2b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:13.051882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddcfa0c-c549-4af5-910b-fea4659c2ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:13.052468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddcfa0c-c549-4af5-910b-fea4659c2ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:13.052757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddcfa0c-c549-4af5-910b-fea4659c2ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:13.053034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddcfa0c-c549-4af5-910b-fea4659c2ab5 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddcfa0c-c549-4af5-910b-fea4659c2ab5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:13.055088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddcfa0c-c549-4af5-910b-fea4659c2ab5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:13.070248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 336/662] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:28:12 2022] GET /identity/v3/services => generated 1258 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:28:13.117444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedade0c-f2d1-44a8-afc5-ff5289f72e0e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:13.168016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:13.191737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:13.192757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:13.193244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:13.193726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:13.194958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:13.195488 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:13.198240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:13.230702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 327/663] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:13.234518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddcfa0c-c549-4af5-910b-fea4659c2ab5 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:13.410528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddcfa0c-c549-4af5-910b-fea4659c2ab5 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:13.477018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:13.478463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:13.479237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:13.479993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:13.497129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:13.500708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:28:13.523774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:13.525534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 00:28:13.527368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:13.531776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 337/664] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Oct 19 00:28:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:13.540951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:13.734109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99207918-c555-457e-b8de-29cd562bc410 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:13.793672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f539ce2-38fc-4a42-97b8-29f0a6c6e88c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:13.794356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f539ce2-38fc-4a42-97b8-29f0a6c6e88c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:13.795260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f539ce2-38fc-4a42-97b8-29f0a6c6e88c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:13.795260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f539ce2-38fc-4a42-97b8-29f0a6c6e88c None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:13.795721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f539ce2-38fc-4a42-97b8-29f0a6c6e88c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:13.796034 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f539ce2-38fc-4a42-97b8-29f0a6c6e88c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:13.797520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f539ce2-38fc-4a42-97b8-29f0a6c6e88c None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:13.817927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 328/665] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Oct 19 00:28:13 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:13.826529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:13.868987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:13.893014 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4289bfa-98cf-4081-a78f-9bff74b98815 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:13.893425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4289bfa-98cf-4081-a78f-9bff74b98815 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:13.893715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4289bfa-98cf-4081-a78f-9bff74b98815 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:13.895420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 329/666] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:13.900357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f89b7aa-f8dc-4704-8a44-c72902df1fd9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:13.900655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f89b7aa-f8dc-4704-8a44-c72902df1fd9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:13.900901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f89b7aa-f8dc-4704-8a44-c72902df1fd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:13.921804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8f89b7aa-f8dc-4704-8a44-c72902df1fd9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:13.923045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8f89b7aa-f8dc-4704-8a44-c72902df1fd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:13.968653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:14.018824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 330/667] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:13 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:14.031800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ec0ff1-b9d7-494b-a143-3766122837cd None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:14.095291 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:14.096378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:14.096857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:14.097333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:14.117818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:14.120466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:14.150669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:14.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:14.152648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:14.152648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:14.152648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:14.155277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 338/668] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:14.160342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f89b7aa-f8dc-4704-8a44-c72902df1fd9 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:14.312974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f89b7aa-f8dc-4704-8a44-c72902df1fd9 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:14.369066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:14.381541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:14.383273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:14.383842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:14.384840 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:14.386039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:14.387063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:14.390297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:14.412788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 331/669] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:14.414821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:14.456758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:14.517187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:14.579571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd51dcca-e72a-4a30-8c69-6dd464bb1a5e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:14.600368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f70b04d-654a-4525-812e-51223f94476e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:14.639058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:14.640029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:14.640392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:14.640734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:14.648814 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:14.650711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:28:14.664811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:14.666600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:14.667844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:28:14.667844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:28:14.667844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:28:14.667844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:28:14.667844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:28:14.667844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:28:14.667844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:14.667844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:14.668587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2de39e0-c0a3-4c9d-80a2-46426b9f390f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:14.668587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de39e0-c0a3-4c9d-80a2-46426b9f390f None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:14.668587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de39e0-c0a3-4c9d-80a2-46426b9f390f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:14.668587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de39e0-c0a3-4c9d-80a2-46426b9f390f None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:14.669066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2de39e0-c0a3-4c9d-80a2-46426b9f390f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:14.670573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 332/670] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:14.670867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2de39e0-c0a3-4c9d-80a2-46426b9f390f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:14.676650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:14.693103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 339/671] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:28:14 2022] POST /identity/v3/services => generated 251 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:28:14.800870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb47c87-64c0-43cc-b48b-432be5daaa1e None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:14.801211 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb47c87-64c0-43cc-b48b-432be5daaa1e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:14.802503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb47c87-64c0-43cc-b48b-432be5daaa1e None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:14.805517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 340/672] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:14.808070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:14.811113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39f87b0-9b5e-41fb-9306-7c5bff59e249 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:14.811469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39f87b0-9b5e-41fb-9306-7c5bff59e249 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:14.811748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39f87b0-9b5e-41fb-9306-7c5bff59e249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:14.822384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdfe6e2-5fc8-4401-8547-4aae4ea2a987 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:14.833474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b39f87b0-9b5e-41fb-9306-7c5bff59e249 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:14.834860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b39f87b0-9b5e-41fb-9306-7c5bff59e249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:14.873839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:14.874915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:14.875408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:14.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:14.876802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:14.877424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:14.879432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:14.894528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 333/673] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:28:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:14.901901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:14.916853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 341/674] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:15.016360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68511bcf-e91d-403e-80a6-bde01971e83c None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:15.055291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d5e6d8-884b-4744-9fdf-b690c10d0d56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:15.056390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d5e6d8-884b-4744-9fdf-b690c10d0d56 None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:15.056878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d5e6d8-884b-4744-9fdf-b690c10d0d56 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:15.057378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d5e6d8-884b-4744-9fdf-b690c10d0d56 None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/f7fcaf24d2ff423a90d9ea6747a67a05/roles/4fcd5d67bffc405288cad5df623aba68` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:15.060538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d5e6d8-884b-4744-9fdf-b690c10d0d56 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=f7fcaf24d2ff423a90d9ea6747a67a05, role_id=4fcd5d67bffc405288cad5df623aba68)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:15.062243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d5e6d8-884b-4744-9fdf-b690c10d0d56 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:15.079557 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 334/675] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:28:14 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/f7fcaf24d2ff423a90d9ea6747a67a05/roles/4fcd5d67bffc405288cad5df623aba68 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:28:15.256548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39f87b0-9b5e-41fb-9306-7c5bff59e249 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:15.329807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:15.379908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39f87b0-9b5e-41fb-9306-7c5bff59e249 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:15.420325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:15.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:15.421781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:15.422265 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:15.423013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:15.424642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:15.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:15.438225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:15.439111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 342/676] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:15 2022] GET /identity/v3/services/object-store => generated 93 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:15.444681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d5e6d8-884b-4744-9fdf-b690c10d0d56 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:15.454594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:15.579181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d5e6d8-884b-4744-9fdf-b690c10d0d56 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:15.639167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:15.640182 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:15.641024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:15.641491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:15.642668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:15.643288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:15.645511 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:15.655602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 335/677] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:15 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:15.664510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:15.849254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6126c904-d799-4f50-a55c-cd34f7099cce None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:15.886195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:15.901872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e690c204-04c6-49b6-b58d-bc250f457a7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:15.902538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e690c204-04c6-49b6-b58d-bc250f457a7f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:15.902826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e690c204-04c6-49b6-b58d-bc250f457a7f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:15.903094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e690c204-04c6-49b6-b58d-bc250f457a7f None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:15.903806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e690c204-04c6-49b6-b58d-bc250f457a7f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:15.904110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e690c204-04c6-49b6-b58d-bc250f457a7f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:15.905526 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e690c204-04c6-49b6-b58d-bc250f457a7f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:15.912101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 343/678] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:15 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:15.918576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:15.930309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd46a6-4127-44e1-88a2-33289827bcf7 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:15.930661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd46a6-4127-44e1-88a2-33289827bcf7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:15.930913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd46a6-4127-44e1-88a2-33289827bcf7 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:15.932602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 344/679] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:15.939748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c651cc-e4cd-4a7f-a785-de8e293d7ad5 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:15.940067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c651cc-e4cd-4a7f-a785-de8e293d7ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:15.940332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c651cc-e4cd-4a7f-a785-de8e293d7ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:15.961093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c2c651cc-e4cd-4a7f-a785-de8e293d7ad5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:15.962462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c2c651cc-e4cd-4a7f-a785-de8e293d7ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:16.015258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:16.054079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdd3a8a-5a93-4232-a234-073855fa8cdc None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:16.096006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:16.096843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:16.097229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:16.097620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:16.098318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 345/680] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:16.099085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] RBAC: Adding query filter params (interface=public, service_id=fa906328b5ef43b9930d371db4330947, region_id=RegionOne) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:16.099562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:16.101197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:16.112141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 336/681] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Oct 19 00:28:15 2022] GET /identity/v3/endpoints?service_id=fa906328b5ef43b9930d371db4330947&interface=public®ion_id=RegionOne => generated 194 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:16.118129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c651cc-e4cd-4a7f-a785-de8e293d7ad5 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:16.256219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c651cc-e4cd-4a7f-a785-de8e293d7ad5 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:16.283055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:16.302411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c623aaa8-e2d5-46aa-8cc2-abf9a82beb9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:16.303473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c623aaa8-e2d5-46aa-8cc2-abf9a82beb9c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:16.303994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c623aaa8-e2d5-46aa-8cc2-abf9a82beb9c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:16.304501 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c623aaa8-e2d5-46aa-8cc2-abf9a82beb9c None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:16.305382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c623aaa8-e2d5-46aa-8cc2-abf9a82beb9c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:16.307109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c623aaa8-e2d5-46aa-8cc2-abf9a82beb9c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:16.315472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 346/682] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:28:16 2022] GET /identity/v3/services => generated 1497 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:28:16.568084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:16.758438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4afda2-9978-4557-90fc-375f9dc96ca7 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:16.823035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:16.824009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:16.824503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:16.824979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:16.826065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:16.828741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:16.832181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:16.842313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:16.843506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 00:28:16.844461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:16.847367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 337/683] 10.222.0.26 () {56 vars in 1132 bytes} [Wed Oct 19 00:28:16 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:16.855023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c623aaa8-e2d5-46aa-8cc2-abf9a82beb9c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:16.974179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c623aaa8-e2d5-46aa-8cc2-abf9a82beb9c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:17.012295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:17.013427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:17.013908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:17.014380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:17.015280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:17.015853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:17.017423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:17.023568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 347/684] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:16 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:17.030202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:17.155078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f202f77-9537-4d0a-904f-75ebf6365a01 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:17.201800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd77515-b375-41e9-b689-fbbf5755bdce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:17.202505 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd77515-b375-41e9-b689-fbbf5755bdce None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:17.202791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd77515-b375-41e9-b689-fbbf5755bdce None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:17.203059 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd77515-b375-41e9-b689-fbbf5755bdce None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:17.203709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd77515-b375-41e9-b689-fbbf5755bdce None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:17.204010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd77515-b375-41e9-b689-fbbf5755bdce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:17.205419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd77515-b375-41e9-b689-fbbf5755bdce None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:17.211571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 338/685] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:17 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:17.217843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:17.276069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:17.301388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b67d7d-5549-4cfb-a9a5-6ab7fd1128cf None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:17.301729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b67d7d-5549-4cfb-a9a5-6ab7fd1128cf None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:17.301938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b67d7d-5549-4cfb-a9a5-6ab7fd1128cf None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:17.303485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 339/686] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:17.308949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb718173-65b6-4fb6-904a-393405b1e4ab None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:17.309223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb718173-65b6-4fb6-904a-393405b1e4ab None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:17.309506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb718173-65b6-4fb6-904a-393405b1e4ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:17.330141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-fb718173-65b6-4fb6-904a-393405b1e4ab None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:17.331625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-fb718173-65b6-4fb6-904a-393405b1e4ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:17.351746 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60aa4749-e8dc-473a-ae31-9537b375b266 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:17.395251 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243dd7cd-62d9-48ff-b37a-e567c1219cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:17.396282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dd7cd-62d9-48ff-b37a-e567c1219cf7 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:17.396765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dd7cd-62d9-48ff-b37a-e567c1219cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:17.397233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dd7cd-62d9-48ff-b37a-e567c1219cf7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:17.398116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243dd7cd-62d9-48ff-b37a-e567c1219cf7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:17.399773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243dd7cd-62d9-48ff-b37a-e567c1219cf7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:17.414635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 340/687] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:17.415027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:17.429244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 348/688] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:28:17 2022] POST /identity/v3/endpoints => generated 347 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:17.463608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f13fe4-937e-4e14-9099-20cccaf4724f None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:17.463926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f13fe4-937e-4e14-9099-20cccaf4724f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:17.464167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f13fe4-937e-4e14-9099-20cccaf4724f None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:17.465653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 341/689] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:17.472275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74925b94-3737-4b78-988c-d8da0e92cb0a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:17.473253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74925b94-3737-4b78-988c-d8da0e92cb0a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:17.473739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74925b94-3737-4b78-988c-d8da0e92cb0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:17.496371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-74925b94-3737-4b78-988c-d8da0e92cb0a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:17.499010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-74925b94-3737-4b78-988c-d8da0e92cb0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:17.605314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 349/690] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:17.705206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f13fe4-937e-4e14-9099-20cccaf4724f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:17.827748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f13fe4-937e-4e14-9099-20cccaf4724f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:17.865917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:17.866578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:17.866855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:17.867122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:17.867765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:17.869202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:17.879423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:17.880482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:17.882069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 342/691] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:17 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:17.883176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74925b94-3737-4b78-988c-d8da0e92cb0a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:17.886602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:17.922375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:17.987483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:18.014281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74925b94-3737-4b78-988c-d8da0e92cb0a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.026434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5208a03-ef81-42ce-8dbb-9e8ecea1e33e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.057487 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ffdb94-af9f-4d82-ac64-ab16a7bc429e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:18.058517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ffdb94-af9f-4d82-ac64-ab16a7bc429e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:18.059026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ffdb94-af9f-4d82-ac64-ab16a7bc429e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:18.059512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ffdb94-af9f-4d82-ac64-ab16a7bc429e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:18.071532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:18.072154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:18.072435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:18.072710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:18.073353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:18.073655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:18.075105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:18.076131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ffdb94-af9f-4d82-ac64-ab16a7bc429e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:28:18.081663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 343/692] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:17 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-55ffdb94-af9f-4d82-ac64-ab16a7bc429e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:28:18.083306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:18.084315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 00:28:18.085272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:18.092043 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:18.092201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 350/693] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Oct 19 00:28:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:18.094112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ffdb94-af9f-4d82-ac64-ab16a7bc429e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:18.226344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:18.246512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ffdb94-af9f-4d82-ac64-ab16a7bc429e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.248222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3adb16-af2b-4241-85d8-8d8b9e466b0d None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.275623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:18.296509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:18.297844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:18.298330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:18.299469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:18.300666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:18.301179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:18.303159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:18.310988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 344/694] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:18 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:18.316582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:18.317541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:18.318685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:18.319216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:18.319751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:18.321002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:18.322467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:18.324369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:18.340186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 351/695] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Oct 19 00:28:18 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:18.348420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:18.456709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3338a0bd-3b8f-44a0-b40e-da1e0ab7ec5b None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.515905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:18.517123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:18.517699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:18.518270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:18.520827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] RBAC: Adding query filter params (interface=public, service_id=4e5d9cf75d3f417fb68606ff23fc2fe5, region_id=RegionOne) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:18.521468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:18.525216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:18.533767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53357443-5ce9-4432-9f04-e962503f46c1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.539987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 345/696] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Oct 19 00:28:18 2022] GET /identity/v3/endpoints?service_id=4e5d9cf75d3f417fb68606ff23fc2fe5&interface=public®ion_id=RegionOne => generated 194 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:18.553753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:18.580519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:18.581512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:18.581943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:18.582381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:18.598309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:18.600120 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:18.620023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:18.621075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:18.621980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:18.621980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:18.621980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:18.629012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 352/697] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:18.630242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:18.634799 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:18.663847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:18.728100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db79b573-4c47-48ea-a242-45e51f8a2459 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.775750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f304c1-58b9-4da7-82c8-4cf3f0fcfcee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:18.777152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f304c1-58b9-4da7-82c8-4cf3f0fcfcee None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:18.777654 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f304c1-58b9-4da7-82c8-4cf3f0fcfcee None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:18.778135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f304c1-58b9-4da7-82c8-4cf3f0fcfcee None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:18.779140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f304c1-58b9-4da7-82c8-4cf3f0fcfcee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:18.780917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f304c1-58b9-4da7-82c8-4cf3f0fcfcee None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:18.782888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7f153b-3393-4a91-8aa6-3ad1d3d5c678 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.789744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 346/698] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:28:18 2022] GET /identity/v3/services => generated 1497 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:28:18.826186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:18.826884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:18.827167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:18.827444 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:18.828125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:18.828429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:18.829856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:18.843724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 353/699] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:18.849914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f304c1-58b9-4da7-82c8-4cf3f0fcfcee None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:18.978183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f304c1-58b9-4da7-82c8-4cf3f0fcfcee None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:18.993291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:19.022783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:19.023420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:19.023698 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:19.023968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:19.032501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:19.033880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:28:19.051435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:19.052542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:19.053740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 347/700] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Oct 19 00:28:18 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:19.062255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:19.137573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:19.180560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dadbdc-1ce8-4809-8548-07ad63d2d2df None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:19.224480 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a33f6d-f965-41b1-937a-66cf3fa96b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:19.225116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a33f6d-f965-41b1-937a-66cf3fa96b94 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:19.225396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a33f6d-f965-41b1-937a-66cf3fa96b94 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:19.225667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a33f6d-f965-41b1-937a-66cf3fa96b94 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:19.226423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a33f6d-f965-41b1-937a-66cf3fa96b94 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:19.226767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a33f6d-f965-41b1-937a-66cf3fa96b94 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:19.228249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a33f6d-f965-41b1-937a-66cf3fa96b94 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:19.249488 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 354/701] 10.222.0.26 () {56 vars in 1149 bytes} [Wed Oct 19 00:28:19 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:19.256049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:19.278592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34982b-17f9-42bd-9ebe-ca6f38001a4a None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:19.279136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34982b-17f9-42bd-9ebe-ca6f38001a4a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:19.279556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34982b-17f9-42bd-9ebe-ca6f38001a4a None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:19.281739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 355/702] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:19.290173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1574418-bb5a-462d-ace2-528f7b41667a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:19.290625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1574418-bb5a-462d-ace2-528f7b41667a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:19.291125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1574418-bb5a-462d-ace2-528f7b41667a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:19.326941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a1574418-bb5a-462d-ace2-528f7b41667a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:19.329050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a1574418-bb5a-462d-ace2-528f7b41667a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:19.392769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fde6ec-3297-427d-bdd2-d1dc7e280cd0 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:19.442509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12d23f3-23e9-4fb6-a71f-dcec6b5a0ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:19.442878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 356/703] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:19.443685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12d23f3-23e9-4fb6-a71f-dcec6b5a0ebc None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:19.444161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12d23f3-23e9-4fb6-a71f-dcec6b5a0ebc None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:19.444632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12d23f3-23e9-4fb6-a71f-dcec6b5a0ebc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:19.507042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:19.527770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12d23f3-23e9-4fb6-a71f-dcec6b5a0ebc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:19.530222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12d23f3-23e9-4fb6-a71f-dcec6b5a0ebc None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:19.664519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 348/704] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:28:19 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 19 00:28:19.671674 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1574418-bb5a-462d-ace2-528f7b41667a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:19.732166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:19.781605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12d23f3-23e9-4fb6-a71f-dcec6b5a0ebc None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:19.795862 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1574418-bb5a-462d-ace2-528f7b41667a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:19.840976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:19.842060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:19.842348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:19.842655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:19.858156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:19.859570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:19.869012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:19.879249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:19.880279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:19.886259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:19.886259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:19.886259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:19.886259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 357/705] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:19.886259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:19.915099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12d23f3-23e9-4fb6-a71f-dcec6b5a0ebc None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:19.959727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:19.960554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:19.960839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:19.961116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:19.961750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:19.963979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:19.978855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:19.979943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 349/706] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:19 2022] GET /identity/v3/services/object-store => generated 93 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:19.987105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:20.037069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a32a79f-6106-43c1-a4ad-81559f78204c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:20.041500 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:20.089960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84284727-be51-454f-9db5-140a8f24284a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:20.091765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84284727-be51-454f-9db5-140a8f24284a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.092272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84284727-be51-454f-9db5-140a8f24284a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.092804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84284727-be51-454f-9db5-140a8f24284a None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.094019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84284727-be51-454f-9db5-140a8f24284a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:20.094697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84284727-be51-454f-9db5-140a8f24284a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:20.097168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84284727-be51-454f-9db5-140a8f24284a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:20.121386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 358/707] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:20.123336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84284727-be51-454f-9db5-140a8f24284a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:20.132535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed24647a-0a3c-4f94-ac94-5af79d6f90df None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:20.181366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f1a4d-5c39-47ff-befb-26a5da534ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:20.181986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f1a4d-5c39-47ff-befb-26a5da534ebe None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.182263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f1a4d-5c39-47ff-befb-26a5da534ebe None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f1a4d-5c39-47ff-befb-26a5da534ebe None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.183272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f1a4d-5c39-47ff-befb-26a5da534ebe None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:20.183635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f1a4d-5c39-47ff-befb-26a5da534ebe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:20.186392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f1a4d-5c39-47ff-befb-26a5da534ebe None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:20.188856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:20.199756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 350/708] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:19 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:20.199756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea80553-2f54-43d4-9ea4-3a0e7de2bc7d None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.199756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea80553-2f54-43d4-9ea4-3a0e7de2bc7d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.199756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea80553-2f54-43d4-9ea4-3a0e7de2bc7d None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.199756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 351/709] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:20.205666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea80553-2f54-43d4-9ea4-3a0e7de2bc7d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:20.269755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84284727-be51-454f-9db5-140a8f24284a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:20.308323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af98400e-adfd-4d61-85fa-e700e87984e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:20.308976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98400e-adfd-4d61-85fa-e700e87984e4 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.309252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98400e-adfd-4d61-85fa-e700e87984e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.310128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98400e-adfd-4d61-85fa-e700e87984e4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.318941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af98400e-adfd-4d61-85fa-e700e87984e4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:20.320417 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af98400e-adfd-4d61-85fa-e700e87984e4 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-af98400e-adfd-4d61-85fa-e700e87984e4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:28:20.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:20.336424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:20.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:28:20.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:28:20.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:28:20.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:28:20.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:28:20.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:28:20.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:20.338151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:20.340634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 359/710] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:20.342186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae3f24f-fc73-465f-8c04-c686d0767ba6 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.342461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae3f24f-fc73-465f-8c04-c686d0767ba6 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.343247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae3f24f-fc73-465f-8c04-c686d0767ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.363174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8ae3f24f-fc73-465f-8c04-c686d0767ba6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:20.364425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8ae3f24f-fc73-465f-8c04-c686d0767ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:20.371506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea80553-2f54-43d4-9ea4-3a0e7de2bc7d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:20.389055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:20.428933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:20.429991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.430599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.431134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.432213 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:20.432740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:20.435421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:20.444370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 352/711] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:20 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:20.445779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:20.470162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 360/712] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:20.471684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae3f24f-fc73-465f-8c04-c686d0767ba6 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:20.537705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:20.629837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae3f24f-fc73-465f-8c04-c686d0767ba6 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:20.631474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c066d93-615f-4b91-8699-80a22acc0ecb None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:20.675736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dfd079-59e6-4f1f-af37-da5d5ba10671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:20.676751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dfd079-59e6-4f1f-af37-da5d5ba10671 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.677207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dfd079-59e6-4f1f-af37-da5d5ba10671 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.677628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dfd079-59e6-4f1f-af37-da5d5ba10671 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.678508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dfd079-59e6-4f1f-af37-da5d5ba10671 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:20.680305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dfd079-59e6-4f1f-af37-da5d5ba10671 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:20.694724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:20.695775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.696134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.696667 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.697780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:20.698284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:20.701705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:20.707724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 361/713] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:28:20 2022] POST /identity/v3/endpoints => generated 349 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:20.724747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 353/714] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:28:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:20.734039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dfd079-59e6-4f1f-af37-da5d5ba10671 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:20.787489 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:20.857010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:20.923588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dfd079-59e6-4f1f-af37-da5d5ba10671 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:20.994108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:20.996195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:20.996195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:20.996195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:20.999849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] RBAC: Adding query filter params (role.id=4fcd5d67bffc405288cad5df623aba68, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=f7fcaf24d2ff423a90d9ea6747a67a05) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:20.999849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:21.001172 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:21.006813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6b256a-8aca-44c0-9153-d5dcab4bf2e7 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:21.014185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 362/715] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:28:20 2022] GET /identity/v3/role_assignments?user.id=f7fcaf24d2ff423a90d9ea6747a67a05&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=4fcd5d67bffc405288cad5df623aba68 => generated 608 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:21.039119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:21.052613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c156233-f8a3-4742-923e-42be29d5f459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:21.053272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c156233-f8a3-4742-923e-42be29d5f459 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:21.053550 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c156233-f8a3-4742-923e-42be29d5f459 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:21.053823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c156233-f8a3-4742-923e-42be29d5f459 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:21.054426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c156233-f8a3-4742-923e-42be29d5f459 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:21.055920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c156233-f8a3-4742-923e-42be29d5f459 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-1c156233-f8a3-4742-923e-42be29d5f459 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:21.067240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:21.068317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:21.069634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 354/716] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:21.081764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:21.216752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:21.266779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4217ed14-95c1-4021-8c1c-2b4867816042 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:21.314160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17825913-2bd4-4eb1-a787-63af1aecc3cb None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:21.314472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17825913-2bd4-4eb1-a787-63af1aecc3cb None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:21.314741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17825913-2bd4-4eb1-a787-63af1aecc3cb None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:21.316368 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 355/717] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:21.321861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41517b1c-a9c0-40cf-898e-73b92cab1eb7 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:21.322192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41517b1c-a9c0-40cf-898e-73b92cab1eb7 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:21.322403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41517b1c-a9c0-40cf-898e-73b92cab1eb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:21.322527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:21.323546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:21.324036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:21.324518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:21.326532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:21.327200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:21.329753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:21.341589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 363/718] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:21 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:21.344279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-41517b1c-a9c0-40cf-898e-73b92cab1eb7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:21.346265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-41517b1c-a9c0-40cf-898e-73b92cab1eb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:21.348324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:21.435489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 356/719] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:21.550605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1de29f-b643-48a9-9f47-b037ce8d7601 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:21.614390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:21.615483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:21.616044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:21.616540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:21.617717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:21.618274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:21.620959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:21.624015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:21.630541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 364/720] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:21 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:21.805672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41517b1c-a9c0-40cf-898e-73b92cab1eb7 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:21.932653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41517b1c-a9c0-40cf-898e-73b92cab1eb7 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:21.953235 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:21.971957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:21.972596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:21.973134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:21.973404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:21.973993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:21.975421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:21.985691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:21.986834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:21.987668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 357/721] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:21 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:21.996590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:22.187114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed7ce7c-cf6c-43f7-8785-aff129013632 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:22.252825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:22.253861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:22.254357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:22.254888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:22.256038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:22.256575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:22.259327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:22.274143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 365/722] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:21 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:22.279810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:22.408567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed27e036-3e29-4ff3-ba3b-00476d54c029 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:22.445695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc4eaf0-adae-451d-9a55-374671294133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:22.446229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4eaf0-adae-451d-9a55-374671294133 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:22.446523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4eaf0-adae-451d-9a55-374671294133 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:22.446806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc4eaf0-adae-451d-9a55-374671294133 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:22.447450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc4eaf0-adae-451d-9a55-374671294133 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:22.447755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc4eaf0-adae-451d-9a55-374671294133 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:22.449128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc4eaf0-adae-451d-9a55-374671294133 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:22.455322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 358/723] 10.222.0.26 () {56 vars in 1114 bytes} [Wed Oct 19 00:28:22 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:22.460916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:22.536652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:22.583477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c70617a-f570-42ed-ae68-7b728aab5be1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:22.627554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9a2333-3ef6-41a0-a561-d26edc5ce0d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:22.628698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9a2333-3ef6-41a0-a561-d26edc5ce0d0 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:22.629208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9a2333-3ef6-41a0-a561-d26edc5ce0d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:22.629705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9a2333-3ef6-41a0-a561-d26edc5ce0d0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:22.631011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9a2333-3ef6-41a0-a561-d26edc5ce0d0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:22.633650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9a2333-3ef6-41a0-a561-d26edc5ce0d0 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:22.674159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 366/724] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:28:22 2022] POST /identity/v3/endpoints => generated 346 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:23.159041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbeecd0-62f3-4f0a-858a-3c16e1a88d0e None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:23.159949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbeecd0-62f3-4f0a-858a-3c16e1a88d0e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:23.160448 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbeecd0-62f3-4f0a-858a-3c16e1a88d0e None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:23.162306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 359/725] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:23.168215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6466b8c7-2c94-46d3-96e9-2cbf25006c61 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:23.168891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6466b8c7-2c94-46d3-96e9-2cbf25006c61 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:23.169372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6466b8c7-2c94-46d3-96e9-2cbf25006c61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:23.190772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-6466b8c7-2c94-46d3-96e9-2cbf25006c61 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:23.192351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6466b8c7-2c94-46d3-96e9-2cbf25006c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:23.305790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 367/726] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:23.460153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac31db2-9fff-4be9-a76f-4211dd69a3c4 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:23.460704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac31db2-9fff-4be9-a76f-4211dd69a3c4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:23.460704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac31db2-9fff-4be9-a76f-4211dd69a3c4 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:23.462318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 360/727] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:23.469007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4a6db9-5580-425e-af8d-24363cc70670 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:23.469587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4a6db9-5580-425e-af8d-24363cc70670 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:23.470082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4a6db9-5580-425e-af8d-24363cc70670 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:23.503764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-2c4a6db9-5580-425e-af8d-24363cc70670 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:23.505983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2c4a6db9-5580-425e-af8d-24363cc70670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:23.588325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:23.639446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 368/728] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:23.677614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac31db2-9fff-4be9-a76f-4211dd69a3c4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:23.734121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:23.795736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac31db2-9fff-4be9-a76f-4211dd69a3c4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:23.832502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:23.833146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:23.833424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:23.833697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:23.834283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:23.835709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:23.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:23.847599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:23.848805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 361/729] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:23 2022] GET /identity/v3/services/object-store => generated 93 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:23.854525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4a6db9-5580-425e-af8d-24363cc70670 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:23.960535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:23.963586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4a6db9-5580-425e-af8d-24363cc70670 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:24.008285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:24.009282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.009780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.010268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.011165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:24.011691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:24.013907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:24.020187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 369/730] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:23 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:24.027210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:24.089187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:24.111623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58845fe-68bb-4569-8892-a3ea1f54e971 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:24.126962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:24.160854 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:24.161525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.161829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.162091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] PATH_INFO: `/v3/services/image` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.162706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:24.164877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:24.177411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:24.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:24.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 362/731] 10.222.0.26 () {56 vars in 1105 bytes} [Wed Oct 19 00:28:23 2022] GET /identity/v3/services/image => generated 86 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:24.193311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:24.214687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c4acb9-6632-46e1-9bcb-29647354ae1f None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:24.268299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:24.268832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.269120 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.269396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.270053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:24.270360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:24.273742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:24.282584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 370/732] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:24 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:24.287871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:24.377501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38755b0b-94d3-47ca-b0b6-2e4831aea610 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:24.433266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818156c0-ca66-44e4-8670-888de012dbba None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:24.435454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:24.445818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469636e1-6713-4ab0-b749-1c5ec1de79d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:24.446882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469636e1-6713-4ab0-b749-1c5ec1de79d4 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.447361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469636e1-6713-4ab0-b749-1c5ec1de79d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.447815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469636e1-6713-4ab0-b749-1c5ec1de79d4 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.448856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469636e1-6713-4ab0-b749-1c5ec1de79d4 None admin] RBAC: Adding query filter params (name=image) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:24.449348 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469636e1-6713-4ab0-b749-1c5ec1de79d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:24.454754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469636e1-6713-4ab0-b749-1c5ec1de79d4 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:24.464063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 363/733] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:24 2022] GET /identity/v3/services?name=image => generated 122 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:24.472035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba75b9e-cfc3-4a9f-9c9a-838486593fce None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.472296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba75b9e-cfc3-4a9f-9c9a-838486593fce None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.472577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba75b9e-cfc3-4a9f-9c9a-838486593fce None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.474153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 364/734] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:24.477163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba75b9e-cfc3-4a9f-9c9a-838486593fce None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:24.484784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f5e165-c6a9-4d55-9233-c318a5468792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:24.485415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f5e165-c6a9-4d55-9233-c318a5468792 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.485695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f5e165-c6a9-4d55-9233-c318a5468792 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.486007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f5e165-c6a9-4d55-9233-c318a5468792 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.486722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f5e165-c6a9-4d55-9233-c318a5468792 None admin] RBAC: Adding query filter params (interface=admin, service_id=fa906328b5ef43b9930d371db4330947, region_id=RegionOne) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:24.487191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f5e165-c6a9-4d55-9233-c318a5468792 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:24.488586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f5e165-c6a9-4d55-9233-c318a5468792 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:24.498121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 371/735] 10.222.0.26 () {56 vars in 1249 bytes} [Wed Oct 19 00:28:24 2022] GET /identity/v3/endpoints?service_id=fa906328b5ef43b9930d371db4330947&interface=admin®ion_id=RegionOne => generated 193 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:24.500097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b686af-9941-4c6d-b43d-e7f2cba0b69c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.500462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b686af-9941-4c6d-b43d-e7f2cba0b69c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.500852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b686af-9941-4c6d-b43d-e7f2cba0b69c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.521138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-15b686af-9941-4c6d-b43d-e7f2cba0b69c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:24.522096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-15b686af-9941-4c6d-b43d-e7f2cba0b69c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:24.555496 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:24.638517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 372/736] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:24.638517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b686af-9941-4c6d-b43d-e7f2cba0b69c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:24.666941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba75b9e-cfc3-4a9f-9c9a-838486593fce None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:24.732659 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:24.733559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.733841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.734088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.735306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] RBAC: Adding query filter params (type=image) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:24.735607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:24.737477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:24.745157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 365/737] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:24 2022] GET /identity/v3/services?type=image => generated 122 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:24.799973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:24.814091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b686af-9941-4c6d-b43d-e7f2cba0b69c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:24.886989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833cf6e4-7850-4194-80b3-7a59bc6f875a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:24.888079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833cf6e4-7850-4194-80b3-7a59bc6f875a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:24.888660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833cf6e4-7850-4194-80b3-7a59bc6f875a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:24.889136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833cf6e4-7850-4194-80b3-7a59bc6f875a None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:24.890309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833cf6e4-7850-4194-80b3-7a59bc6f875a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:24.893062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833cf6e4-7850-4194-80b3-7a59bc6f875a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:24.907066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 373/738] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:28:24 2022] GET /identity/v3/services => generated 1497 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:28:25.027244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:25.165351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00583c1a-c259-4d7d-9511-90d3cc2304c1 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:25.167316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:25.189643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6186de5d-6d9e-4916-9b9d-5a2c041093dd None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:25.190004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6186de5d-6d9e-4916-9b9d-5a2c041093dd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:25.190200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6186de5d-6d9e-4916-9b9d-5a2c041093dd None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:25.191876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 374/739] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:25.197503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace1870a-f4df-4c3d-9a00-8e31b69a0032 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:25.197803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace1870a-f4df-4c3d-9a00-8e31b69a0032 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:25.198053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace1870a-f4df-4c3d-9a00-8e31b69a0032 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:25.219252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ace1870a-f4df-4c3d-9a00-8e31b69a0032 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:25.220452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ace1870a-f4df-4c3d-9a00-8e31b69a0032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:25.222171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0373290c-4de1-46b1-9934-e05595e83f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:25.223470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0373290c-4de1-46b1-9934-e05595e83f39 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:25.223753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0373290c-4de1-46b1-9934-e05595e83f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:25.224024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0373290c-4de1-46b1-9934-e05595e83f39 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:25.225746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0373290c-4de1-46b1-9934-e05595e83f39 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:25.227175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0373290c-4de1-46b1-9934-e05595e83f39 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:25.250582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 366/740] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:28:25 2022] POST /identity/v3/services => generated 242 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:25.331974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 375/741] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:25 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:25.631590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0373290c-4de1-46b1-9934-e05595e83f39 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:25.746142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0373290c-4de1-46b1-9934-e05595e83f39 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:25.783610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:25.784226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:25.784504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:25.784775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:25.785439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:25.786932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:25.789623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING keystone.server.flask.common [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 19 00:28:25.808927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.sql.core [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 00:28:25.808927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 00:28:25.808927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [parameters: {'id': 'f021091e5c804d05bcefe3c99a181902', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 19 00:28:25.808927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81812) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f021091e5c804d05bcefe3c99a181902.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f021091e5c804d05bcefe3c99a181902. Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:28:25.817060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 00:28:25.818097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application yield resource Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise self.value Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application session.commit() Oct 19 00:28:25.819419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.flush() Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application compat.raise_( Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 00:28:25.820544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 00:28:25.821704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:28:25.823783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [parameters: {'id': 'f021091e5c804d05bcefe3c99a181902', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:25.827004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:25.832240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f021091e5c804d05bcefe3c99a181902. Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:25.838952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 367/742] 10.222.0.26 () {60 vars in 1140 bytes} [Wed Oct 19 00:28:25 2022] POST /identity/v3/projects => generated 262 bytes in 188 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:25.840248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace1870a-f4df-4c3d-9a00-8e31b69a0032 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:25.938395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:25.955774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace1870a-f4df-4c3d-9a00-8e31b69a0032 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:25.998136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a89e71-0845-4bc3-b980-7b3b5e195aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:25.998970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a89e71-0845-4bc3-b980-7b3b5e195aff None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:25.999247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a89e71-0845-4bc3-b980-7b3b5e195aff None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:25.999517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a89e71-0845-4bc3-b980-7b3b5e195aff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:26.007525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a89e71-0845-4bc3-b980-7b3b5e195aff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:26.008997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a89e71-0845-4bc3-b980-7b3b5e195aff None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-62a89e71-0845-4bc3-b980-7b3b5e195aff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:28:26.022477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:26.024592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:26.025803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 376/743] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:26.030676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:26.055361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:26.188480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ba1821-ef4d-4eda-a5ad-b5880ffe9681 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:26.253950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3aa7d8-bb89-408d-bb28-1b64120ff45a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:26.255427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3aa7d8-bb89-408d-bb28-1b64120ff45a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:26.256137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3aa7d8-bb89-408d-bb28-1b64120ff45a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:26.256870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3aa7d8-bb89-408d-bb28-1b64120ff45a None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:26.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3aa7d8-bb89-408d-bb28-1b64120ff45a None admin] RBAC: Adding query filter params (name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:26.259243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3aa7d8-bb89-408d-bb28-1b64120ff45a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:26.262708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3aa7d8-bb89-408d-bb28-1b64120ff45a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:26.305151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 368/744] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:26 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:27.103573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faf33d2-ece5-45e1-aa2a-8a260be76508 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:27.104267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faf33d2-ece5-45e1-aa2a-8a260be76508 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:27.104267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faf33d2-ece5-45e1-aa2a-8a260be76508 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:27.105809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 377/745] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:27.111800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b46717-5a3e-49ab-8e66-330f5e7b61f2 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:27.112400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b46717-5a3e-49ab-8e66-330f5e7b61f2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:27.112400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b46717-5a3e-49ab-8e66-330f5e7b61f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:27.132379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e5b46717-5a3e-49ab-8e66-330f5e7b61f2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:27.133948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e5b46717-5a3e-49ab-8e66-330f5e7b61f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:27.256788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 369/746] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:27 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:27.333945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047589d9-2604-4877-b04f-cf8bc37c26fe None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:27.334297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047589d9-2604-4877-b04f-cf8bc37c26fe None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:27.334478 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047589d9-2604-4877-b04f-cf8bc37c26fe None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:27.336121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 378/747] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:27.342252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2f5b4-d698-4152-8a12-9f98d898c7f5 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:27.342565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2f5b4-d698-4152-8a12-9f98d898c7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:27.342835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2f5b4-d698-4152-8a12-9f98d898c7f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:27.366003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-3be2f5b4-d698-4152-8a12-9f98d898c7f5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:27.367903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3be2f5b4-d698-4152-8a12-9f98d898c7f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:27.370631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:27.463333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 370/748] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:27 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:27.548683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047589d9-2604-4877-b04f-cf8bc37c26fe None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:27.616709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e49b80-8f89-4970-8af8-d2b631bcfbcd None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:27.617017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e49b80-8f89-4970-8af8-d2b631bcfbcd None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:27.617236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e49b80-8f89-4970-8af8-d2b631bcfbcd None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:27.618859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 371/749] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:27.625077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcc7858-cd13-4b58-84c7-4c1d8fa66710 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:27.625370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcc7858-cd13-4b58-84c7-4c1d8fa66710 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:27.625637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcc7858-cd13-4b58-84c7-4c1d8fa66710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:27.648857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-4fcc7858-cd13-4b58-84c7-4c1d8fa66710 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:27.650366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4fcc7858-cd13-4b58-84c7-4c1d8fa66710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:27.672622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047589d9-2604-4877-b04f-cf8bc37c26fe None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:27.725198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2432855-1a71-4ca6-902d-53f7129794a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:27.726194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2432855-1a71-4ca6-902d-53f7129794a3 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:27.726720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2432855-1a71-4ca6-902d-53f7129794a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:27.727204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2432855-1a71-4ca6-902d-53f7129794a3 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:27.728343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2432855-1a71-4ca6-902d-53f7129794a3 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:27.730988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2432855-1a71-4ca6-902d-53f7129794a3 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:27.756046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 372/750] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:27 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:27.756046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcc7858-cd13-4b58-84c7-4c1d8fa66710 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:27.759757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 379/751] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:28:27 2022] POST /identity/v3/services => generated 242 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:27.886406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:27.953933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcc7858-cd13-4b58-84c7-4c1d8fa66710 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:28.012830 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:28.013983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.014265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.014616 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.015233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:28.016810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:28.027015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:28.028683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:28.029722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 373/752] 10.222.0.26 () {56 vars in 1126 bytes} [Wed Oct 19 00:28:27 2022] GET /identity/v3/services/object-store => generated 93 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:28.037189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2432855-1a71-4ca6-902d-53f7129794a3 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:28.047823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:28.050876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:28.177826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2432855-1a71-4ca6-902d-53f7129794a3 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:28.208519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f2db3-8831-4f3d-a692-6500e4eefee4 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:28.227625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:28.229736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.230268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.231055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.232860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:28.233739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:28.235983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:28.249244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 380/753] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:28 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:28.252714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:28.253199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:28.262838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:28.263669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.264156 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.264659 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] PATH_INFO: `/v3/services/compute` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.265499 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:28.267206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:28.280115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:28.281475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:28.282779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:28.290433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 374/754] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:28 2022] GET /identity/v3/services/compute => generated 88 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:28.291721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:28.376545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:28.423701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ed9df9-fc0e-46b9-8588-7fb0840af798 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:28.429197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2f30db-f0e0-4336-abe3-172f7f61261e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:28.468098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:28.469078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.469588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.470107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.471200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:28.471732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:28.473624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:28.479195 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:28.479815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.480091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.480355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.480990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:28.481367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:28.483194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:28.483662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 381/755] 10.222.0.26 () {56 vars in 1122 bytes} [Wed Oct 19 00:28:28 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:28.489727 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:28.502340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 375/756] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:28 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:28.509201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:28.605870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:28.629428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54eb025-6aaf-4faf-a9a9-6bad64cfb78c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:28.669837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe76882-a63e-4bf1-be4c-1f50feccc27c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:28.670784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe76882-a63e-4bf1-be4c-1f50feccc27c None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.671154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe76882-a63e-4bf1-be4c-1f50feccc27c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.671432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe76882-a63e-4bf1-be4c-1f50feccc27c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.672124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe76882-a63e-4bf1-be4c-1f50feccc27c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:28.673527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe76882-a63e-4bf1-be4c-1f50feccc27c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:28.679409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:28.681807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33972e7f-d36e-463c-afb6-4c5dabe7680a None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:28.696632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 382/757] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:28:28 2022] POST /identity/v3/endpoints => generated 325 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:28.735711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09590597-c3f9-46ca-ba3b-0d41903a738a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:28.736441 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09590597-c3f9-46ca-ba3b-0d41903a738a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.736727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09590597-c3f9-46ca-ba3b-0d41903a738a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.737002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09590597-c3f9-46ca-ba3b-0d41903a738a None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.737647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09590597-c3f9-46ca-ba3b-0d41903a738a None admin] RBAC: Adding query filter params (type=compute) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:28.737945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09590597-c3f9-46ca-ba3b-0d41903a738a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:28.739409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09590597-c3f9-46ca-ba3b-0d41903a738a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:28.750635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 376/758] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:28 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:28.752348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe76882-a63e-4bf1-be4c-1f50feccc27c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:28.888602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe76882-a63e-4bf1-be4c-1f50feccc27c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:28.920883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5c63c-a00c-4bcd-92c7-e1de37a28d89 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.921243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5c63c-a00c-4bcd-92c7-e1de37a28d89 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.921434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5c63c-a00c-4bcd-92c7-e1de37a28d89 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.923346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 377/759] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:28.928752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f537322d-e153-4257-8f62-9d8de1f29366 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.929045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f537322d-e153-4257-8f62-9d8de1f29366 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.929306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f537322d-e153-4257-8f62-9d8de1f29366 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.933659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:28.935162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:28.935460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:28.935702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:28.936414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] RBAC: Adding query filter params (interface=public, service_id=e554f7c3aa6b42deb07ff01d4387302d, region_id=RegionOne) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:28.936722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:28.939486 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:28.944343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 383/760] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Oct 19 00:28:28 2022] GET /identity/v3/endpoints?service_id=e554f7c3aa6b42deb07ff01d4387302d&interface=public®ion_id=RegionOne => generated 194 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:28.952080 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:28.958075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-f537322d-e153-4257-8f62-9d8de1f29366 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:28.959370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f537322d-e153-4257-8f62-9d8de1f29366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:29.037335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:29.066099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 378/761] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:28 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:29.108780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d3fe66-d91c-4d63-a806-f77c125776c6 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:29.150922 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0729d5cd-3ba1-411c-ae12-2be39582d8e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:29.151580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0729d5cd-3ba1-411c-ae12-2be39582d8e9 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:29.151851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0729d5cd-3ba1-411c-ae12-2be39582d8e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:29.152127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0729d5cd-3ba1-411c-ae12-2be39582d8e9 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:29.152752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0729d5cd-3ba1-411c-ae12-2be39582d8e9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:29.155395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0729d5cd-3ba1-411c-ae12-2be39582d8e9 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:29.170701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 384/762] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:28:28 2022] GET /identity/v3/services => generated 1957 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:28:29.346223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f537322d-e153-4257-8f62-9d8de1f29366 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:29.439329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:29.545921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f537322d-e153-4257-8f62-9d8de1f29366 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:29.623543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:29.624506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:29.625009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:29.625513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:29.626769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:29.629555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:29.654970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.sql.core [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 00:28:29.654970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 19 00:28:29.654970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [parameters: {'user_id': 'f86f55e812144ee596dd690f9502d66e', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 19 00:28:29.654970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81812) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:28:29.670582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:28:29.672355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application yield resource Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise self.value Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application session.commit() Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 00:28:29.674141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.flush() Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application compat.raise_( Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 00:28:29.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:28:29.677624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:28:29.682900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [parameters: {'user_id': 'f86f55e812144ee596dd690f9502d66e', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:29.683823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 19 00:28:29.685252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 19 00:28:29.686178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 19 00:28:29.686178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 19 00:28:29.686178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 00:28:29.686178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 00:28:29.686178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 19 00:28:29.686178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:29.686178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 379/763] 10.222.0.26 () {60 vars in 1131 bytes} [Wed Oct 19 00:28:29 2022] POST /identity/v3/users => generated 160 bytes in 328 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:29.686653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0729d5cd-3ba1-411c-ae12-2be39582d8e9 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:29.807782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0729d5cd-3ba1-411c-ae12-2be39582d8e9 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:29.834067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:29.854995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b794d5eb-d5be-4843-871e-c970e373692f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:29.856000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b794d5eb-d5be-4843-871e-c970e373692f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:29.857005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b794d5eb-d5be-4843-871e-c970e373692f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:29.857466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b794d5eb-d5be-4843-871e-c970e373692f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:29.871940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b794d5eb-d5be-4843-871e-c970e373692f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:29.874726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b794d5eb-d5be-4843-871e-c970e373692f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-b794d5eb-d5be-4843-871e-c970e373692f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:28:29.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:29.898872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 00:28:29.899828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:29.900398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 385/764] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:28:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:29.907482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:30.078311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea1687-b9f4-4d68-9fb2-3adf10f5893a None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:30.096718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:30.146792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3f10c6-6d83-4c9d-9dbd-037c162a5513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:30.147686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3f10c6-6d83-4c9d-9dbd-037c162a5513 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:30.148103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3f10c6-6d83-4c9d-9dbd-037c162a5513 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:30.148544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3f10c6-6d83-4c9d-9dbd-037c162a5513 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:30.149828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3f10c6-6d83-4c9d-9dbd-037c162a5513 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:30.150329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3f10c6-6d83-4c9d-9dbd-037c162a5513 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:30.153613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3f10c6-6d83-4c9d-9dbd-037c162a5513 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:30.230108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 380/765] 10.222.0.26 () {56 vars in 1101 bytes} [Wed Oct 19 00:28:29 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:30.811828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f355d6e1-a29f-4285-98d7-46f7436c480c None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:30.812315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f355d6e1-a29f-4285-98d7-46f7436c480c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:30.812706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f355d6e1-a29f-4285-98d7-46f7436c480c None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:30.815218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 386/766] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:30.821178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5aac5-c52f-490f-b4a2-428fa698aaa3 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:30.821948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5aac5-c52f-490f-b4a2-428fa698aaa3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:30.821948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5aac5-c52f-490f-b4a2-428fa698aaa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:30.850125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-32b5aac5-c52f-490f-b4a2-428fa698aaa3 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:30.852923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-32b5aac5-c52f-490f-b4a2-428fa698aaa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:30.975353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 381/767] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:30 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:31.249469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f355d6e1-a29f-4285-98d7-46f7436c480c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:31.376115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f355d6e1-a29f-4285-98d7-46f7436c480c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:31.420147 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd7fa7f-abc0-490d-b6c3-ac188cd91d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:31.420832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd7fa7f-abc0-490d-b6c3-ac188cd91d34 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:31.421089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd7fa7f-abc0-490d-b6c3-ac188cd91d34 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:31.421352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd7fa7f-abc0-490d-b6c3-ac188cd91d34 None admin] PATH_INFO: `/v3/services/image` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:31.421956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd7fa7f-abc0-490d-b6c3-ac188cd91d34 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:31.423423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd7fa7f-abc0-490d-b6c3-ac188cd91d34 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-4fd7fa7f-abc0-490d-b6c3-ac188cd91d34 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:31.433567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:31.436049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:31.437703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:31.440558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 387/768] 10.222.0.26 () {56 vars in 1105 bytes} [Wed Oct 19 00:28:31 2022] GET /identity/v3/services/image => generated 86 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:31.441481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b5aac5-c52f-490f-b4a2-428fa698aaa3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:31.530044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b12d0-96c7-403a-9ab2-4fc723f3ae97 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:31.530375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b12d0-96c7-403a-9ab2-4fc723f3ae97 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:31.530783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b12d0-96c7-403a-9ab2-4fc723f3ae97 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:31.532658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 388/769] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:31.539222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ac527c-7224-4db8-9d53-39af5f149652 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:31.539633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ac527c-7224-4db8-9d53-39af5f149652 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:31.540091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ac527c-7224-4db8-9d53-39af5f149652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:31.566999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-49ac527c-7224-4db8-9d53-39af5f149652 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:31.568433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-49ac527c-7224-4db8-9d53-39af5f149652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:31.581983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:31.589839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b5aac5-c52f-490f-b4a2-428fa698aaa3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:31.635626 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:31.639629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:31.640704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:31.641092 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:31.641536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:31.642657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] RBAC: Adding query filter params (name=image) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:31.643152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:31.645642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:31.659327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 382/770] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:31 2022] GET /identity/v3/services?name=image => generated 122 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:31.661395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:31.686910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 389/771] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:31 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:31.847298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb35239-703e-45ea-b545-d292d8411bf3 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:31.915258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2a4a30-f52c-4575-a6d1-7d5e8692d0e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:31.916238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2a4a30-f52c-4575-a6d1-7d5e8692d0e0 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:31.916677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2a4a30-f52c-4575-a6d1-7d5e8692d0e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:31.917115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2a4a30-f52c-4575-a6d1-7d5e8692d0e0 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:31.918215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2a4a30-f52c-4575-a6d1-7d5e8692d0e0 None admin] RBAC: Adding query filter params (type=image) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:31.918772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2a4a30-f52c-4575-a6d1-7d5e8692d0e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:31.921253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2a4a30-f52c-4575-a6d1-7d5e8692d0e0 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:31.936230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 383/772] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:31 2022] GET /identity/v3/services?type=image => generated 350 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:31.937521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ac527c-7224-4db8-9d53-39af5f149652 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:32.009896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8791d-409a-4bf1-b29b-492001b9a515 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:32.010948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8791d-409a-4bf1-b29b-492001b9a515 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:32.011556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8791d-409a-4bf1-b29b-492001b9a515 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:32.014941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 384/773] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:32 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:32.020823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866b2dc3-8db2-47e6-a89e-23ef6343004f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:32.021160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866b2dc3-8db2-47e6-a89e-23ef6343004f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:32.021378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866b2dc3-8db2-47e6-a89e-23ef6343004f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:32.042987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-866b2dc3-8db2-47e6-a89e-23ef6343004f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:32.045485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-866b2dc3-8db2-47e6-a89e-23ef6343004f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:32.067563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ac527c-7224-4db8-9d53-39af5f149652 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:32.069386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:32.125869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:32.126383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:32.126694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:32.126968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:32.127691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] RBAC: Adding query filter params (interface=public, service_id=ed29adff28fe43c5a28249aa6a767d10, region_id=RegionOne) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:32.128010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:32.129401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:32.135161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 390/774] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Oct 19 00:28:31 2022] GET /identity/v3/endpoints?service_id=ed29adff28fe43c5a28249aa6a767d10&interface=public®ion_id=RegionOne => generated 194 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:32.135986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:32.140363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 385/775] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:32 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:32.144209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866b2dc3-8db2-47e6-a89e-23ef6343004f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:32.205419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:32.297410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866b2dc3-8db2-47e6-a89e-23ef6343004f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:32.302796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0516779a-07fa-44c2-bea7-ebdc2733e196 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:32.355124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284b86c-355f-40ef-8444-17dec3aa9f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:32.356025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284b86c-355f-40ef-8444-17dec3aa9f35 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:32.356485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284b86c-355f-40ef-8444-17dec3aa9f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:32.356960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284b86c-355f-40ef-8444-17dec3aa9f35 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:32.357985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284b86c-355f-40ef-8444-17dec3aa9f35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:32.360429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284b86c-355f-40ef-8444-17dec3aa9f35 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:32.362621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5f3304-119a-4334-91d6-56520431c21a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:32.364063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5f3304-119a-4334-91d6-56520431c21a None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:32.364598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5f3304-119a-4334-91d6-56520431c21a None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:32.365118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5f3304-119a-4334-91d6-56520431c21a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:32.366979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5f3304-119a-4334-91d6-56520431c21a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:32.369443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5f3304-119a-4334-91d6-56520431c21a None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:32.373347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 386/777] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:28:32 2022] GET /identity/v3/services => generated 1957 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:28:32.373480 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 391/777] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:32.382243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5f3304-119a-4334-91d6-56520431c21a None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:32.426366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284b86c-355f-40ef-8444-17dec3aa9f35 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:32.447637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:32.550393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284b86c-355f-40ef-8444-17dec3aa9f35 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:32.577739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5f3304-119a-4334-91d6-56520431c21a None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:32.590157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:32.603798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:32.604454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:32.604734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:32.605007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] PATH_INFO: `/v3/services/compute` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:32.605603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:32.608820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:32.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:32.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:32.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 387/778] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:32 2022] GET /identity/v3/services/compute => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:32.627536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:32.630771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f58d1b-f1e9-4dfa-8394-e1942351dafe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:32.631803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f58d1b-f1e9-4dfa-8394-e1942351dafe None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:32.631916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f58d1b-f1e9-4dfa-8394-e1942351dafe None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:32.632126 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f58d1b-f1e9-4dfa-8394-e1942351dafe None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:32.633281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f58d1b-f1e9-4dfa-8394-e1942351dafe None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:32.634263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f58d1b-f1e9-4dfa-8394-e1942351dafe None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:32.672298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 392/779] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:28:32 2022] POST /identity/v3/projects => generated 311 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:32.773018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:32.810346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23203304-6e22-4c0f-b8cd-3ec69c9d468b None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:32.877774 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:32.879734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:32.880244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:32.880734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:32.881985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:32.882569 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:32.885306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:32.902534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 388/780] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:32 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:32.906185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f58d1b-f1e9-4dfa-8394-e1942351dafe None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:33.100830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f58d1b-f1e9-4dfa-8394-e1942351dafe None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:33.167011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ff30f2-d619-4d1a-b737-b5cccdce85cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:33.167865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff30f2-d619-4d1a-b737-b5cccdce85cb None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:33.168303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff30f2-d619-4d1a-b737-b5cccdce85cb None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:33.168766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ff30f2-d619-4d1a-b737-b5cccdce85cb None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:33.169872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ff30f2-d619-4d1a-b737-b5cccdce85cb None admin] RBAC: Adding query filter params (type=compute) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:33.170346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ff30f2-d619-4d1a-b737-b5cccdce85cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:33.172901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ff30f2-d619-4d1a-b737-b5cccdce85cb None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:33.190043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 393/781] 10.222.0.26 () {56 vars in 1112 bytes} [Wed Oct 19 00:28:32 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:33.191589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:33.387266 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59aff35-5105-417b-b79e-c4edd3916568 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:33.454904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eae940e-9b35-4c2c-8851-78feba186798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:33.455890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eae940e-9b35-4c2c-8851-78feba186798 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:33.456341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eae940e-9b35-4c2c-8851-78feba186798 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:33.456795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eae940e-9b35-4c2c-8851-78feba186798 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:33.458038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eae940e-9b35-4c2c-8851-78feba186798 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:33.461885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eae940e-9b35-4c2c-8851-78feba186798 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:33.510525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 389/782] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:28:33 2022] POST /identity/v3/endpoints => generated 334 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:34.767640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35506f7-3a5e-423d-aba3-e1cae13d9187 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:34.768185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35506f7-3a5e-423d-aba3-e1cae13d9187 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:34.768624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35506f7-3a5e-423d-aba3-e1cae13d9187 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:34.771500 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 394/783] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:34.777575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef1037-e9c8-452c-a383-aad1ab3b11a6 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:34.779336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef1037-e9c8-452c-a383-aad1ab3b11a6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:34.779336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef1037-e9c8-452c-a383-aad1ab3b11a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:34.786188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:34.801766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-45ef1037-e9c8-452c-a383-aad1ab3b11a6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:34.803760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-45ef1037-e9c8-452c-a383-aad1ab3b11a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:34.903261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 390/784] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:35.180275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35506f7-3a5e-423d-aba3-e1cae13d9187 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:35.229429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:35.379477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35506f7-3a5e-423d-aba3-e1cae13d9187 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:35.436062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:35.437026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:35.437524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:35.437973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] PATH_INFO: `/v3/services/image` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:35.438812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:35.439259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3b5a7c-0c47-458a-8e0a-4605fa52c306 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:35.439554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3b5a7c-0c47-458a-8e0a-4605fa52c306 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:35.439834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3b5a7c-0c47-458a-8e0a-4605fa52c306 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:35.441044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:35.442519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 391/785] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:35.447272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893aca6-4890-4551-b26f-95a687ad1db4 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:35.447599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893aca6-4890-4551-b26f-95a687ad1db4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:35.447976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893aca6-4890-4551-b26f-95a687ad1db4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:35.452839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:35.455425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:35.457017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 395/786] 10.222.0.26 () {56 vars in 1105 bytes} [Wed Oct 19 00:28:35 2022] GET /identity/v3/services/image => generated 86 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:35.460210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:35.473119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2893aca6-4890-4551-b26f-95a687ad1db4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:35.474365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2893aca6-4890-4551-b26f-95a687ad1db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:35.576697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 392/787] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:35 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:35.607306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:35.630231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6927e511-8fab-4637-bed6-0b47dd233dba None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:35.691692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:35.692581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:35.693065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:35.693543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:35.694807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] RBAC: Adding query filter params (name=image) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:35.695347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:35.697909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:35.708180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 396/788] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:35 2022] GET /identity/v3/services?name=image => generated 122 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:35.717774 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2893aca6-4890-4551-b26f-95a687ad1db4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:35.820656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:35.874124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2893aca6-4890-4551-b26f-95a687ad1db4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:35.912414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:35.913022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:35.913296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:35.913562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:35.914189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] RBAC: Adding query filter params (type=image) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:35.914513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:35.915896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:35.922387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 393/789] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:35 2022] GET /identity/v3/services?type=image => generated 350 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:35.928874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:36.074302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:36.118733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6c4e80-bf48-417a-bff2-8012f0eee774 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:36.188261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358c748b-0593-4ba7-be9b-2c602c64072f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:36.189320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358c748b-0593-4ba7-be9b-2c602c64072f None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:36.189808 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358c748b-0593-4ba7-be9b-2c602c64072f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:36.190297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358c748b-0593-4ba7-be9b-2c602c64072f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:36.191612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358c748b-0593-4ba7-be9b-2c602c64072f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:36.194214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358c748b-0593-4ba7-be9b-2c602c64072f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:36.200538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07a0b2d-0ed5-4a90-8f87-6ea821a724d7 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:36.231779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:36.247020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 397/790] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:28:35 2022] POST /identity/v3/endpoints => generated 327 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:36.272317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1efefd6-237f-4556-8350-2a7fc8f0f260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:36.273495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1efefd6-237f-4556-8350-2a7fc8f0f260 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:36.273988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1efefd6-237f-4556-8350-2a7fc8f0f260 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:36.274474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1efefd6-237f-4556-8350-2a7fc8f0f260 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:36.276689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1efefd6-237f-4556-8350-2a7fc8f0f260 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:36.280157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1efefd6-237f-4556-8350-2a7fc8f0f260 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:36.286546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 394/791] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:36.290118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358c748b-0593-4ba7-be9b-2c602c64072f None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:36.306275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:36.432426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358c748b-0593-4ba7-be9b-2c602c64072f None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:36.477800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d82f5-30f4-4dff-8bfa-8bd8bc77d098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:36.478661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d82f5-30f4-4dff-8bfa-8bd8bc77d098 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:36.479038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d82f5-30f4-4dff-8bfa-8bd8bc77d098 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:36.479406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d82f5-30f4-4dff-8bfa-8bd8bc77d098 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:36.480178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d82f5-30f4-4dff-8bfa-8bd8bc77d098 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:36.481944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d82f5-30f4-4dff-8bfa-8bd8bc77d098 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:36.518537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 398/792] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:28:36 2022] POST /identity/v3/users => generated 285 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:28:37.039670 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef122b2a-03c8-4d61-9bbe-06e3214176cb None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:37.040459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef122b2a-03c8-4d61-9bbe-06e3214176cb None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:37.040846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef122b2a-03c8-4d61-9bbe-06e3214176cb None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:37.043684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 395/793] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:37 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:37.052860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9665e38-ce08-42de-9cd5-20eda27ec9b5 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:37.053679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9665e38-ce08-42de-9cd5-20eda27ec9b5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:37.053830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9665e38-ce08-42de-9cd5-20eda27ec9b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:37.084070 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c9665e38-ce08-42de-9cd5-20eda27ec9b5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:37.085963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c9665e38-ce08-42de-9cd5-20eda27ec9b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:37.200816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:37.219204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 399/794] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:37.596404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef122b2a-03c8-4d61-9bbe-06e3214176cb None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:37.778602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef122b2a-03c8-4d61-9bbe-06e3214176cb None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:37.849717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047ecaa-5337-420e-8e06-20deef5d99a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:37.850709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047ecaa-5337-420e-8e06-20deef5d99a1 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:37.851191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047ecaa-5337-420e-8e06-20deef5d99a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:37.851673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047ecaa-5337-420e-8e06-20deef5d99a1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:37.880768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a047ecaa-5337-420e-8e06-20deef5d99a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-a047ecaa-5337-420e-8e06-20deef5d99a1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:28:37.890959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:37.892786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:37.894469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 396/795] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Oct 19 00:28:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:37.903170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9665e38-ce08-42de-9cd5-20eda27ec9b5 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:38.100430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9665e38-ce08-42de-9cd5-20eda27ec9b5 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:38.162474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:38.163561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:38.164056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:38.164544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:38.165771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:38.166318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:38.169565 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:38.185356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 400/796] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Oct 19 00:28:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:38.193860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047ecaa-5337-420e-8e06-20deef5d99a1 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:38.252869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:38.328399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047ecaa-5337-420e-8e06-20deef5d99a1 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:38.372139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:38.372914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:38.373193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:38.373464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:38.385498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:38.387562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:38.405013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:38.414986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:38.415990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:38.415990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:38.415990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:38.415990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 397/797] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:38.416432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:38.531400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:38.601626 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59799bbb-699d-4867-a1f9-87eac4ee0bc3 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:38.644765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:38.645292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:38.645563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:38.645837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:38.646521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:38.646812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:38.648245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:38.664100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 401/798] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:38.670186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:38.791286 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:38.797009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4825c-6ce8-4450-8c0b-e6ef8a397e8b None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:38.838126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2a4e0c-8416-4f41-b796-0053b8e2bb89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:38.838992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2a4e0c-8416-4f41-b796-0053b8e2bb89 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:38.839407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2a4e0c-8416-4f41-b796-0053b8e2bb89 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:38.839806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2a4e0c-8416-4f41-b796-0053b8e2bb89 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:38.850136 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2a4e0c-8416-4f41-b796-0053b8e2bb89 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:38.851859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2a4e0c-8416-4f41-b796-0053b8e2bb89 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-5b2a4e0c-8416-4f41-b796-0053b8e2bb89 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:28:38.866609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:38.868431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:28:38.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:38.873765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 398/799] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Oct 19 00:28:38 2022] GET /identity/v3/users/nova => generated 82 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:38.876904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:39.056360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5d9ac9-4668-4fa4-a37b-77a62c763915 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:39.088914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:39.122289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:39.123423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.123908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.124405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.125658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:39.126164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:39.128777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:39.144428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5e0c7a-cd71-4200-bc9a-85256565cdf4 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.144831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5e0c7a-cd71-4200-bc9a-85256565cdf4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.145215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5e0c7a-cd71-4200-bc9a-85256565cdf4 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.147193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 399/800] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:39.156461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf3081-dbbc-4d04-b8f7-ca2bd8ad3a52 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.156972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf3081-dbbc-4d04-b8f7-ca2bd8ad3a52 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.157347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf3081-dbbc-4d04-b8f7-ca2bd8ad3a52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.168844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 402/801] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Oct 19 00:28:38 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:39.170474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:39.188087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-f5bf3081-dbbc-4d04-b8f7-ca2bd8ad3a52 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:39.190056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f5bf3081-dbbc-4d04-b8f7-ca2bd8ad3a52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:39.311076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 400/802] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:39.354112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9efbc7-7873-4447-a9a2-19759b7e33e1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:39.399830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:39.400478 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.400755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.401031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.415282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:39.416808 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:39.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:39.443459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:39.445213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:39.445213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:39.445213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:39.445213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 403/803] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:39.450210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bf3081-dbbc-4d04-b8f7-ca2bd8ad3a52 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:39.581307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:39.599226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bf3081-dbbc-4d04-b8f7-ca2bd8ad3a52 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:39.606303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:39.639340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:39.645378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9fff5b-7e3c-4f9e-8ad5-78e80857fbff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:39.646002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9fff5b-7e3c-4f9e-8ad5-78e80857fbff None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.646297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9fff5b-7e3c-4f9e-8ad5-78e80857fbff None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.646572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9fff5b-7e3c-4f9e-8ad5-78e80857fbff None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.647260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9fff5b-7e3c-4f9e-8ad5-78e80857fbff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:39.647566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9fff5b-7e3c-4f9e-8ad5-78e80857fbff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:39.648983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9fff5b-7e3c-4f9e-8ad5-78e80857fbff None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:39.664972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 401/804] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:39.671715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bee8a3-3f09-42c1-9ed5-e08a6734c80a None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.671999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bee8a3-3f09-42c1-9ed5-e08a6734c80a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.672275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bee8a3-3f09-42c1-9ed5-e08a6734c80a None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.673648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 402/805] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:39.674517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bee8a3-3f09-42c1-9ed5-e08a6734c80a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:39.743626 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdd1925-25e1-4e33-a2e1-aa7357b0cfdb None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:39.783606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c24a06b-b15a-4541-98c6-a54375e04e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:39.784379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c24a06b-b15a-4541-98c6-a54375e04e8f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.784747 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c24a06b-b15a-4541-98c6-a54375e04e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.785112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c24a06b-b15a-4541-98c6-a54375e04e8f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.798955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c24a06b-b15a-4541-98c6-a54375e04e8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-8c24a06b-b15a-4541-98c6-a54375e04e8f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:28:39.803909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:39.805733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 404/806] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:28:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:39.807418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79316911-fa26-42f4-a011-ad7498bd2ff8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.808870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79316911-fa26-42f4-a011-ad7498bd2ff8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.808870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79316911-fa26-42f4-a011-ad7498bd2ff8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.828249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-79316911-fa26-42f4-a011-ad7498bd2ff8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:39.829490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-79316911-fa26-42f4-a011-ad7498bd2ff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:39.830665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bee8a3-3f09-42c1-9ed5-e08a6734c80a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:39.890561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:39.891511 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:39.891993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:39.892478 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:39.904963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:39.906449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:39.908204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 405/807] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:39.909740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79316911-fa26-42f4-a011-ad7498bd2ff8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:28:39.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:39.920527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:39.921460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 403/808] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:39.926623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:39.940457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:39.952822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:40.064294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85250db4-c75b-46d2-b038-459155dcff7e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.078371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79316911-fa26-42f4-a011-ad7498bd2ff8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.118358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.119210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.119490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.119880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.120776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:40.121320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.123436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:40.129161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.129933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.130400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.130860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.131991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:40.132503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.135047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:40.140658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 404/809] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:28:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:40.146400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:40.148756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 406/810] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:28:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:40.160706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:40.229988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:40.269735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:40.276086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72ed409-7700-42d2-a01f-78f776584494 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.295918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6488ed58-fa90-43d4-bace-3eb8132db039 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.314107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0dec12-43cc-4d6d-af93-f034fb855e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.314800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0dec12-43cc-4d6d-af93-f034fb855e5c None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.315074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0dec12-43cc-4d6d-af93-f034fb855e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.315344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0dec12-43cc-4d6d-af93-f034fb855e5c None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/41009aec3b1843aa80323c374cc01461/roles/4fcd5d67bffc405288cad5df623aba68` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.317494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0dec12-43cc-4d6d-af93-f034fb855e5c None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=41009aec3b1843aa80323c374cc01461, role_id=4fcd5d67bffc405288cad5df623aba68)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.318954 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0dec12-43cc-4d6d-af93-f034fb855e5c None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:40.334371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 405/811] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:28:40 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/41009aec3b1843aa80323c374cc01461/roles/4fcd5d67bffc405288cad5df623aba68 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:28:40.335214 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0dec12-43cc-4d6d-af93-f034fb855e5c None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:40.336264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7497cf3-9ab4-497d-b778-ef7241fed0ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.336923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7497cf3-9ab4-497d-b778-ef7241fed0ef None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.337200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7497cf3-9ab4-497d-b778-ef7241fed0ef None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.337465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7497cf3-9ab4-497d-b778-ef7241fed0ef None admin] PATH_INFO: `/v3/users/5abfec9a85fc479f8376064c33da261c` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.361848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7497cf3-9ab4-497d-b778-ef7241fed0ef None admin] RBAC: Authorizing `identity:get_user(user_id=5abfec9a85fc479f8376064c33da261c)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.363486 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7497cf3-9ab4-497d-b778-ef7241fed0ef None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:40.365249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 407/812] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:28:40 2022] GET /identity/v3/users/5abfec9a85fc479f8376064c33da261c => generated 285 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:40.372278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7497cf3-9ab4-497d-b778-ef7241fed0ef None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:40.462414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0dec12-43cc-4d6d-af93-f034fb855e5c None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.495774 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:40.508392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.509021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.509645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.509645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.510241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.511724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:40.521126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.sql.core [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 00:28:40.521126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 00:28:40.521126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [parameters: {'id': 'f976dfc4b8654f9f9001c306259b4c83', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 00:28:40.521126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81812) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:28:40.530407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 00:28:40.532165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application yield resource Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application next(self.gen) Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise self.value Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application session.commit() Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 00:28:40.534013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.session.flush() Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application compat.raise_( Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 00:28:40.535806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 00:28:40.537675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application conn.query(q) Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application result.read() Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 00:28:40.539771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [parameters: {'id': 'f976dfc4b8654f9f9001c306259b4c83', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 00:28:40.548211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:40.556622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 00:28:40.560694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:40.562575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7497cf3-9ab4-497d-b778-ef7241fed0ef None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.562755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 406/813] 10.222.0.26 () {60 vars in 1137 bytes} [Wed Oct 19 00:28:40 2022] POST /identity/v3/domains => generated 190 bytes in 202 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:40.562755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:40.593684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc3be4a-d7ed-4bc1-bb57-6845daaf310e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.594381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc3be4a-d7ed-4bc1-bb57-6845daaf310e None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.594741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc3be4a-d7ed-4bc1-bb57-6845daaf310e None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.595018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc3be4a-d7ed-4bc1-bb57-6845daaf310e None admin] PATH_INFO: `/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.596505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc3be4a-d7ed-4bc1-bb57-6845daaf310e None admin] RBAC: Authorizing `identity:get_project(project_id=1106304e80ca46cc8404cfdd2dbcbf07)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.598288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc3be4a-d7ed-4bc1-bb57-6845daaf310e None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:40.600091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 408/814] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:28:40 2022] GET /identity/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07 => generated 311 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:40.606540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc3be4a-d7ed-4bc1-bb57-6845daaf310e None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:40.611144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:40.699053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f743ac7-0659-4b73-96e6-14761979a85d None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.752888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.753677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.754093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.754517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.755579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc3be4a-d7ed-4bc1-bb57-6845daaf310e None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.767800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.769754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:40.792289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:40.794206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:40.795469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:40.795469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:40.795469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:40.799481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 407/815] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:40.800283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:40.812170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03bf2ba-5ae0-4cea-bdc3-4b6cbcfc4171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.812842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03bf2ba-5ae0-4cea-bdc3-4b6cbcfc4171 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.813158 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03bf2ba-5ae0-4cea-bdc3-4b6cbcfc4171 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.813390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03bf2ba-5ae0-4cea-bdc3-4b6cbcfc4171 None admin] PATH_INFO: `/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07/users/5abfec9a85fc479f8376064c33da261c/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.815431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03bf2ba-5ae0-4cea-bdc3-4b6cbcfc4171 None admin] RBAC: Authorizing `identity:create_grant(project_id=1106304e80ca46cc8404cfdd2dbcbf07, user_id=5abfec9a85fc479f8376064c33da261c, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.816913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03bf2ba-5ae0-4cea-bdc3-4b6cbcfc4171 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:40.839356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 409/816] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:28:40 2022] PUT /identity/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07/users/5abfec9a85fc479f8376064c33da261c/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:28:40.864120 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:40.945964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6024b98-1679-47d4-ab4e-70d49118cdf9 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:40.987396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747c9d27-62ad-44bb-bba5-50be6f30b5f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:40.988022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c9d27-62ad-44bb-bba5-50be6f30b5f2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:40.988305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c9d27-62ad-44bb-bba5-50be6f30b5f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:40.988580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c9d27-62ad-44bb-bba5-50be6f30b5f2 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:40.989251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747c9d27-62ad-44bb-bba5-50be6f30b5f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:40.989557 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747c9d27-62ad-44bb-bba5-50be6f30b5f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:40.991062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747c9d27-62ad-44bb-bba5-50be6f30b5f2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:41.007046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 408/817] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:42.594420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582f86f5-d42d-48a9-b752-76dc48a3aa86 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:42.595013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582f86f5-d42d-48a9-b752-76dc48a3aa86 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:42.595013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582f86f5-d42d-48a9-b752-76dc48a3aa86 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:42.596325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 410/818] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:42.602042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0c9364-df74-4e19-8661-c03582868b45 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:42.602518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0c9364-df74-4e19-8661-c03582868b45 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:42.602730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0c9364-df74-4e19-8661-c03582868b45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:42.623548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-bd0c9364-df74-4e19-8661-c03582868b45 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:42.624775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bd0c9364-df74-4e19-8661-c03582868b45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:42.696635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 409/819] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:42.977790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582f86f5-d42d-48a9-b752-76dc48a3aa86 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:43.052389 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba5f228-ced3-417d-bce3-1d4c21638295 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.052671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba5f228-ced3-417d-bce3-1d4c21638295 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.052941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba5f228-ced3-417d-bce3-1d4c21638295 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:43.053764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:43.054317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 410/820] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:43.060501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c348bc57-e186-49ca-ba65-15d680a2cf01 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.060773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c348bc57-e186-49ca-ba65-15d680a2cf01 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.061052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c348bc57-e186-49ca-ba65-15d680a2cf01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:43.080642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c348bc57-e186-49ca-ba65-15d680a2cf01 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:43.081845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c348bc57-e186-49ca-ba65-15d680a2cf01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:43.105568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582f86f5-d42d-48a9-b752-76dc48a3aa86 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:43.113180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:43.148843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145327f3-7cd0-4236-b3a3-ce71bea8e8a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:43.149490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145327f3-7cd0-4236-b3a3-ce71bea8e8a7 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.149768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145327f3-7cd0-4236-b3a3-ce71bea8e8a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.150044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145327f3-7cd0-4236-b3a3-ce71bea8e8a7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:43.160008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 411/821] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:43.162284 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145327f3-7cd0-4236-b3a3-ce71bea8e8a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-145327f3-7cd0-4236-b3a3-ce71bea8e8a7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:28:43.167244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:43.168874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:43.169846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 411/822] 10.222.0.26 () {56 vars in 1120 bytes} [Wed Oct 19 00:28:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:43.175373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c348bc57-e186-49ca-ba65-15d680a2cf01 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:43.285984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c348bc57-e186-49ca-ba65-15d680a2cf01 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:43.322706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:43.323317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.323880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.324157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:43.325962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:43.326260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:43.327690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:43.337018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 412/823] 10.222.0.26 () {56 vars in 1115 bytes} [Wed Oct 19 00:28:43 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:43.344321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145327f3-7cd0-4236-b3a3-ce71bea8e8a7 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:43.456871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:43.487533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:43.500121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145327f3-7cd0-4236-b3a3-ce71bea8e8a7 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:43.530930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:43.547966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:43.548956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.549524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.550066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:43.562587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:43.564087 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:43.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:43.582317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:43.583322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:43.583322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:43.583322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:43.583498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab9fad-a23e-45c6-8d1d-a4dfbaa96ef8 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:43.583584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 412/824] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:28:43.587765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:43.625722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd3b58e-c6f0-4242-a9d6-d6efeb702943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:43.626680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3b58e-c6f0-4242-a9d6-d6efeb702943 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.626680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3b58e-c6f0-4242-a9d6-d6efeb702943 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.627092 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd3b58e-c6f0-4242-a9d6-d6efeb702943 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:43.639545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd3b58e-c6f0-4242-a9d6-d6efeb702943 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-ebd3b58e-c6f0-4242-a9d6-d6efeb702943 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:28:43.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:43.646324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:28:43.647930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 413/825] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:28:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:43.653251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd3b58e-c6f0-4242-a9d6-d6efeb702943 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:43.722989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12cb94-61c0-407b-9568-360f304cc1b1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:43.778415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:43.778866 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:43.779623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.780112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.780529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:43.781412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:43.781804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:43.783647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:43.797940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd3b58e-c6f0-4242-a9d6-d6efeb702943 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:43.798962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 413/826] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:43.805419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:43.839406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:43.840286 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.840740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.841102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:43.841887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:43.842288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:43.844242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:43.848283 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:43.854511 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 414/827] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:28:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:43.862010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:43.942660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80976059-b8e5-45d2-8405-377e6a500d78 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:43.991496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:43.992497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:43.992947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:43.993373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:44.005998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:44.007230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e99a5-6fc7-4444-983c-54f981cc8539 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:44.008496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:28:44.029907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:44.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:44.032855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 414/828] 10.222.0.26 () {56 vars in 1093 bytes} [Wed Oct 19 00:28:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:44.040797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:44.063567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef952b6-a9c8-483e-9708-3760f968e935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:44.064477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef952b6-a9c8-483e-9708-3760f968e935 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:44.064928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef952b6-a9c8-483e-9708-3760f968e935 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:44.065325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef952b6-a9c8-483e-9708-3760f968e935 None admin] PATH_INFO: `/v3/users/5abfec9a85fc479f8376064c33da261c` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:44.067005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef952b6-a9c8-483e-9708-3760f968e935 None admin] RBAC: Authorizing `identity:get_user(user_id=5abfec9a85fc479f8376064c33da261c)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:44.069471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef952b6-a9c8-483e-9708-3760f968e935 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:44.072293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 415/829] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:28:43 2022] GET /identity/v3/users/5abfec9a85fc479f8376064c33da261c => generated 285 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:44.079661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef952b6-a9c8-483e-9708-3760f968e935 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:44.110564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:44.181321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:44.188407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c60733-1a65-45f4-9717-c6feaf8ea146 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:44.220090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef952b6-a9c8-483e-9708-3760f968e935 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:44.226408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:44.227058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:44.227340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:44.227616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:44.228354 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:44.228662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:44.230111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:44.249679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 415/830] 10.222.0.26 () {56 vars in 1133 bytes} [Wed Oct 19 00:28:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:44.255636 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:44.265228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb8a62-0127-428b-b333-ca5a128c923a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:44.266005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb8a62-0127-428b-b333-ca5a128c923a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:44.266393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb8a62-0127-428b-b333-ca5a128c923a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:44.266844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb8a62-0127-428b-b333-ca5a128c923a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:44.359245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcb8a62-0127-428b-b333-ca5a128c923a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:44.361808 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcb8a62-0127-428b-b333-ca5a128c923a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:44.367762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b9607-13ee-49e0-9fc7-03bad8959554 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:44.393700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:44.405930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2292f935-22d3-4ee7-96af-c7554d84d96c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:44.407045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292f935-22d3-4ee7-96af-c7554d84d96c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:44.407550 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292f935-22d3-4ee7-96af-c7554d84d96c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:44.408046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292f935-22d3-4ee7-96af-c7554d84d96c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:44.455119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:44.504827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 416/831] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:28:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 19 00:28:44.506507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2292f935-22d3-4ee7-96af-c7554d84d96c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:44.507998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2292f935-22d3-4ee7-96af-c7554d84d96c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:44.511616 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb8a62-0127-428b-b333-ca5a128c923a None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:44.672435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb8a62-0127-428b-b333-ca5a128c923a None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:44.693302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:44.716401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d540a5e-bb41-441b-a3fc-792579e93b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:44.717174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d540a5e-bb41-441b-a3fc-792579e93b87 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:44.717554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d540a5e-bb41-441b-a3fc-792579e93b87 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:44.717908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d540a5e-bb41-441b-a3fc-792579e93b87 None admin] PATH_INFO: `/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:44.719190 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d540a5e-bb41-441b-a3fc-792579e93b87 None admin] RBAC: Authorizing `identity:get_project(project_id=1106304e80ca46cc8404cfdd2dbcbf07)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:44.720787 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d540a5e-bb41-441b-a3fc-792579e93b87 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:44.723070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 417/832] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:28:44 2022] GET /identity/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07 => generated 311 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:44.729283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d540a5e-bb41-441b-a3fc-792579e93b87 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:44.746878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 416/833] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:28:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 19 00:28:44.753081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2292f935-22d3-4ee7-96af-c7554d84d96c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:44.817272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:44.857112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d540a5e-bb41-441b-a3fc-792579e93b87 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:44.882176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2292f935-22d3-4ee7-96af-c7554d84d96c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:44.896242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:44.896925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:44.897270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:44.897663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:44.898558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=1106304e80ca46cc8404cfdd2dbcbf07, user.id=5abfec9a85fc479f8376064c33da261c) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:44.899040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:44.900679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:44.908460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 418/834] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:28:44 2022] GET /identity/v3/role_assignments?user.id=5abfec9a85fc479f8376064c33da261c&scope.project.id=1106304e80ca46cc8404cfdd2dbcbf07&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:44.923603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:44.924156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:44.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:44.924694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:44.937955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:44.939530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:28:44.957741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:44.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:28:44.961008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:28:44.961008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:28:44.961008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:44.961008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 417/835] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:44.965833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:45.032465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:45.219158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a2037e-ad02-4ee3-9464-076fe21ab8a5 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:45.255484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:45.256084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:45.256357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:45.256625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:45.257260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:45.257555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:45.258947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:45.270742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 419/836] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:28:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:45.276661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:45.394746 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e398ad-8eac-46bd-92a3-88309e45dabf None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:45.404338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:45.442896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:45.443535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:45.443820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:45.444154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:45.451661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:45.453121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:28:45.466436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:28:45.468363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:28:45.470206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 418/837] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:28:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:28:45.474507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:45.605321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e9c129-7d2d-4f98-bdd6-2b733c50074a None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:45.616045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:45.645897 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbecee2-7891-4c04-8986-8b52308ef38c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:45.646614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbecee2-7891-4c04-8986-8b52308ef38c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:45.646951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbecee2-7891-4c04-8986-8b52308ef38c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:45.647263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbecee2-7891-4c04-8986-8b52308ef38c None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:45.648015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbecee2-7891-4c04-8986-8b52308ef38c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:45.648358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbecee2-7891-4c04-8986-8b52308ef38c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:45.649945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbecee2-7891-4c04-8986-8b52308ef38c None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:45.664204 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 420/838] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:28:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:45.671111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:45.814142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c85594-5875-4afc-aa8b-92d5fea799c8 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:45.874128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023cbfb9-af56-41f8-beb9-6c36e17dcd7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:45.875214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cbfb9-af56-41f8-beb9-6c36e17dcd7c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:45.875711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cbfb9-af56-41f8-beb9-6c36e17dcd7c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:45.876200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cbfb9-af56-41f8-beb9-6c36e17dcd7c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:45.877599 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023cbfb9-af56-41f8-beb9-6c36e17dcd7c None admin] RBAC: Adding query filter params (role.id=4fcd5d67bffc405288cad5df623aba68, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=41009aec3b1843aa80323c374cc01461) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:45.878164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023cbfb9-af56-41f8-beb9-6c36e17dcd7c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:45.880810 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023cbfb9-af56-41f8-beb9-6c36e17dcd7c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:45.892604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 419/839] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:28:45 2022] GET /identity/v3/role_assignments?user.id=41009aec3b1843aa80323c374cc01461&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=4fcd5d67bffc405288cad5df623aba68 => generated 608 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:47.270515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f0e0a-f109-43e5-ae23-79bf5d8c2965 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:47.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f0e0a-f109-43e5-ae23-79bf5d8c2965 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:47.271220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f0e0a-f109-43e5-ae23-79bf5d8c2965 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:47.272483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 421/840] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:47.282085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d0ba5-671b-4e56-a577-b5cb5adb5ce9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:47.282801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d0ba5-671b-4e56-a577-b5cb5adb5ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:47.283080 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d0ba5-671b-4e56-a577-b5cb5adb5ce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:47.314064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e17d0ba5-671b-4e56-a577-b5cb5adb5ce9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:47.316127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e17d0ba5-671b-4e56-a577-b5cb5adb5ce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:47.348697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:47.403647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 420/841] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:47.687100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90f0e0a-f109-43e5-ae23-79bf5d8c2965 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:47.870819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90f0e0a-f109-43e5-ae23-79bf5d8c2965 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:47.932680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eb4ca1-fcf0-492d-9a08-2adde1c29f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:47.934682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb4ca1-fcf0-492d-9a08-2adde1c29f3b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:47.935080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb4ca1-fcf0-492d-9a08-2adde1c29f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:47.935406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb4ca1-fcf0-492d-9a08-2adde1c29f3b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:47.936754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eb4ca1-fcf0-492d-9a08-2adde1c29f3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:47.938272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eb4ca1-fcf0-492d-9a08-2adde1c29f3b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:47.940003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 422/842] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:47.946037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17d0ba5-671b-4e56-a577-b5cb5adb5ce9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:48.053851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17d0ba5-671b-4e56-a577-b5cb5adb5ce9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:48.094889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3c6a7a-e0a2-4ed0-9af3-c4f27e042b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:48.095499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3c6a7a-e0a2-4ed0-9af3-c4f27e042b38 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:48.095776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3c6a7a-e0a2-4ed0-9af3-c4f27e042b38 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:48.096044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3c6a7a-e0a2-4ed0-9af3-c4f27e042b38 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:48.096705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3c6a7a-e0a2-4ed0-9af3-c4f27e042b38 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:48.098426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3c6a7a-e0a2-4ed0-9af3-c4f27e042b38 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:48.123004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 421/843] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:28:47 2022] POST /identity/v3/users => generated 286 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:28:50.368371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79390b-1aef-4d91-b3ea-72d69245ad9f None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:50.368964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79390b-1aef-4d91-b3ea-72d69245ad9f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:50.368964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79390b-1aef-4d91-b3ea-72d69245ad9f None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:50.370359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 423/844] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:50.376324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c3a09-8ca5-46d8-a658-c5f310de4d0b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:50.376827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c3a09-8ca5-46d8-a658-c5f310de4d0b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:50.376971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c3a09-8ca5-46d8-a658-c5f310de4d0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:50.396368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8b5c3a09-8ca5-46d8-a658-c5f310de4d0b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:50.397743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8b5c3a09-8ca5-46d8-a658-c5f310de4d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:50.409982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:50.470744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 422/845] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:28:50.820781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb79390b-1aef-4d91-b3ea-72d69245ad9f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:50.861365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:50.934904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb79390b-1aef-4d91-b3ea-72d69245ad9f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:50.977188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c92538-4b0d-4acd-98c1-f774c07bafbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:50.977896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c92538-4b0d-4acd-98c1-f774c07bafbf None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:50.978216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c92538-4b0d-4acd-98c1-f774c07bafbf None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:50.978573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c92538-4b0d-4acd-98c1-f774c07bafbf None admin] PATH_INFO: `/v3/roles/66029780d52042b9a44109eb91c0d80e` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:50.980180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c92538-4b0d-4acd-98c1-f774c07bafbf None admin] RBAC: Authorizing `identity:get_role(role_id=66029780d52042b9a44109eb91c0d80e)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:50.982426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c92538-4b0d-4acd-98c1-f774c07bafbf None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:50.983918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 424/846] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:28:50 2022] GET /identity/v3/roles/66029780d52042b9a44109eb91c0d80e => generated 223 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:50.989977 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5c3a09-8ca5-46d8-a658-c5f310de4d0b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:51.096777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5c3a09-8ca5-46d8-a658-c5f310de4d0b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:51.138096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab90e3c-1c0e-4515-bf63-0ca54d97a0c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:51.138786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab90e3c-1c0e-4515-bf63-0ca54d97a0c1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:51.139062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab90e3c-1c0e-4515-bf63-0ca54d97a0c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:51.139333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab90e3c-1c0e-4515-bf63-0ca54d97a0c1 None admin] PATH_INFO: `/v3/users/5af8478cc0044e99a054a857239af78b` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:51.154593 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab90e3c-1c0e-4515-bf63-0ca54d97a0c1 None admin] RBAC: Authorizing `identity:get_user(user_id=5af8478cc0044e99a054a857239af78b)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:51.155989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab90e3c-1c0e-4515-bf63-0ca54d97a0c1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:51.157774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 423/847] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:28:50 2022] GET /identity/v3/users/5af8478cc0044e99a054a857239af78b => generated 286 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:51.163740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c92538-4b0d-4acd-98c1-f774c07bafbf None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:51.273631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c92538-4b0d-4acd-98c1-f774c07bafbf None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:51.276052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:51.309231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20db2b05-137e-4e7f-916b-7d91b200331d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:51.309871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db2b05-137e-4e7f-916b-7d91b200331d None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:51.310150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db2b05-137e-4e7f-916b-7d91b200331d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:51.310418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db2b05-137e-4e7f-916b-7d91b200331d None admin] PATH_INFO: `/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:51.311585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20db2b05-137e-4e7f-916b-7d91b200331d None admin] RBAC: Authorizing `identity:get_project(project_id=1106304e80ca46cc8404cfdd2dbcbf07)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:51.313018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20db2b05-137e-4e7f-916b-7d91b200331d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:51.314694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 425/848] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:28:51 2022] GET /identity/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07 => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:51.320701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab90e3c-1c0e-4515-bf63-0ca54d97a0c1 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:51.325322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:51.429266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab90e3c-1c0e-4515-bf63-0ca54d97a0c1 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:51.467983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4221cf4e-a928-426c-9daa-fd03895f08b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:51.468657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4221cf4e-a928-426c-9daa-fd03895f08b7 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:51.468931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4221cf4e-a928-426c-9daa-fd03895f08b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:51.469203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4221cf4e-a928-426c-9daa-fd03895f08b7 None admin] PATH_INFO: `/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07/users/5af8478cc0044e99a054a857239af78b/roles/66029780d52042b9a44109eb91c0d80e` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:51.471381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4221cf4e-a928-426c-9daa-fd03895f08b7 None admin] RBAC: Authorizing `identity:create_grant(project_id=1106304e80ca46cc8404cfdd2dbcbf07, user_id=5af8478cc0044e99a054a857239af78b, role_id=66029780d52042b9a44109eb91c0d80e)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:51.473516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4221cf4e-a928-426c-9daa-fd03895f08b7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:51.491556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 424/849] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:28:51 2022] PUT /identity/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07/users/5af8478cc0044e99a054a857239af78b/roles/66029780d52042b9a44109eb91c0d80e => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:28:53.696071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebf14ea-d0c4-45e2-8db3-06b31136fe56 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:53.696795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebf14ea-d0c4-45e2-8db3-06b31136fe56 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:53.696795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebf14ea-d0c4-45e2-8db3-06b31136fe56 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:53.698302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 426/850] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:28:53.705437 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c97b5a-a368-40ee-a83a-b73274b906c9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:53.706034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c97b5a-a368-40ee-a83a-b73274b906c9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:53.706034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c97b5a-a368-40ee-a83a-b73274b906c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:53.726829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e9c97b5a-a368-40ee-a83a-b73274b906c9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:53.728180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e9c97b5a-a368-40ee-a83a-b73274b906c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:53.796115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:53.808032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 425/851] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:54.077933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebf14ea-d0c4-45e2-8db3-06b31136fe56 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:54.190613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebf14ea-d0c4-45e2-8db3-06b31136fe56 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:54.231347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c34221c-5b33-41f4-b0ca-83d409457ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:54.232001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c34221c-5b33-41f4-b0ca-83d409457ecf None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:54.232272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c34221c-5b33-41f4-b0ca-83d409457ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:54.232536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c34221c-5b33-41f4-b0ca-83d409457ecf None admin] PATH_INFO: `/v3/roles/66029780d52042b9a44109eb91c0d80e` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:54.234078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c34221c-5b33-41f4-b0ca-83d409457ecf None admin] RBAC: Authorizing `identity:get_role(role_id=66029780d52042b9a44109eb91c0d80e)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:54.235610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c34221c-5b33-41f4-b0ca-83d409457ecf None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:54.237032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 427/852] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:28:54 2022] GET /identity/v3/roles/66029780d52042b9a44109eb91c0d80e => generated 223 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:54.243286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c97b5a-a368-40ee-a83a-b73274b906c9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:54.358800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c97b5a-a368-40ee-a83a-b73274b906c9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:54.398420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd18789-0308-4e1f-a9bf-3fb539b8b2d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:54.399179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd18789-0308-4e1f-a9bf-3fb539b8b2d7 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:54.399499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd18789-0308-4e1f-a9bf-3fb539b8b2d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:54.399812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd18789-0308-4e1f-a9bf-3fb539b8b2d7 None admin] PATH_INFO: `/v3/users/5af8478cc0044e99a054a857239af78b` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:54.401110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd18789-0308-4e1f-a9bf-3fb539b8b2d7 None admin] RBAC: Authorizing `identity:get_user(user_id=5af8478cc0044e99a054a857239af78b)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:54.403212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd18789-0308-4e1f-a9bf-3fb539b8b2d7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:54.404966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 426/853] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:28:54 2022] GET /identity/v3/users/5af8478cc0044e99a054a857239af78b => generated 286 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:54.411267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c34221c-5b33-41f4-b0ca-83d409457ecf None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:54.461122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:54.526334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c34221c-5b33-41f4-b0ca-83d409457ecf None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:54.565084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb8af43-209e-4f47-8e73-67d823ebc01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:54.565692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8af43-209e-4f47-8e73-67d823ebc01d None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:54.565965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8af43-209e-4f47-8e73-67d823ebc01d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:54.566232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8af43-209e-4f47-8e73-67d823ebc01d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:54.643310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb8af43-209e-4f47-8e73-67d823ebc01d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:54.644718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb8af43-209e-4f47-8e73-67d823ebc01d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:54.691784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:54.759396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 428/854] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:28:54 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 00:28:54.766101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd18789-0308-4e1f-a9bf-3fb539b8b2d7 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:54.837103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:54.886243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd18789-0308-4e1f-a9bf-3fb539b8b2d7 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:54.923878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e14802-126c-434a-bc46-c4c37c2054cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:54.924436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e14802-126c-434a-bc46-c4c37c2054cc None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:54.924716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e14802-126c-434a-bc46-c4c37c2054cc None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:54.924981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e14802-126c-434a-bc46-c4c37c2054cc None admin] PATH_INFO: `/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:54.926118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e14802-126c-434a-bc46-c4c37c2054cc None admin] RBAC: Authorizing `identity:get_project(project_id=1106304e80ca46cc8404cfdd2dbcbf07)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:54.927574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e14802-126c-434a-bc46-c4c37c2054cc None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:54.929214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 427/855] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:28:54 2022] GET /identity/v3/projects/1106304e80ca46cc8404cfdd2dbcbf07 => generated 311 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:54.936542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb8af43-209e-4f47-8e73-67d823ebc01d None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:55.045624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb8af43-209e-4f47-8e73-67d823ebc01d None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:55.082421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf58d81-2b4c-43cd-9dc6-4b4718b8ed98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:55.083046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf58d81-2b4c-43cd-9dc6-4b4718b8ed98 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:55.083319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf58d81-2b4c-43cd-9dc6-4b4718b8ed98 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:55.083585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf58d81-2b4c-43cd-9dc6-4b4718b8ed98 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:55.084308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf58d81-2b4c-43cd-9dc6-4b4718b8ed98 None admin] RBAC: Adding query filter params (role.id=66029780d52042b9a44109eb91c0d80e, scope.project.id=1106304e80ca46cc8404cfdd2dbcbf07, user.id=5af8478cc0044e99a054a857239af78b) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:28:55.084610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf58d81-2b4c-43cd-9dc6-4b4718b8ed98 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:55.086041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf58d81-2b4c-43cd-9dc6-4b4718b8ed98 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:55.092498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 429/856] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:28:54 2022] GET /identity/v3/role_assignments?user.id=5af8478cc0044e99a054a857239af78b&scope.project.id=1106304e80ca46cc8404cfdd2dbcbf07&role.id=66029780d52042b9a44109eb91c0d80e => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:28:57.320386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6530ce-939c-4961-bbe6-71dcdfafb482 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:57.320959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6530ce-939c-4961-bbe6-71dcdfafb482 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:57.320959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6530ce-939c-4961-bbe6-71dcdfafb482 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:57.322258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 428/857] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:28:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:28:57.328748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec57b294-5580-4c2f-9a87-94654dd374ca None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:57.329280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec57b294-5580-4c2f-9a87-94654dd374ca None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:57.329280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec57b294-5580-4c2f-9a87-94654dd374ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:57.336684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:57.349381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ec57b294-5580-4c2f-9a87-94654dd374ca None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:28:57.350640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ec57b294-5580-4c2f-9a87-94654dd374ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:28:57.421840 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 430/858] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:28:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:28:57.698670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6530ce-939c-4961-bbe6-71dcdfafb482 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:57.812467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6530ce-939c-4961-bbe6-71dcdfafb482 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:57.845409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:57.852437 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f77b452-bab5-4f78-a9b5-95159c2002fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:57.853082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77b452-bab5-4f78-a9b5-95159c2002fc None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:57.853363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77b452-bab5-4f78-a9b5-95159c2002fc None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:57.853635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77b452-bab5-4f78-a9b5-95159c2002fc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:57.854841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f77b452-bab5-4f78-a9b5-95159c2002fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:57.856368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f77b452-bab5-4f78-a9b5-95159c2002fc None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:57.857988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 429/859] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:28:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:28:57.864182 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec57b294-5580-4c2f-9a87-94654dd374ca None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:28:57.975698 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec57b294-5580-4c2f-9a87-94654dd374ca None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:28:58.015308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135941f3-ffc7-4939-aa53-8b6c4d5c699b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:28:58.015930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135941f3-ffc7-4939-aa53-8b6c4d5c699b None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:28:58.016209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135941f3-ffc7-4939-aa53-8b6c4d5c699b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:28:58.016481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135941f3-ffc7-4939-aa53-8b6c4d5c699b None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:28:58.017151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135941f3-ffc7-4939-aa53-8b6c4d5c699b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:28:58.018635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135941f3-ffc7-4939-aa53-8b6c4d5c699b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:28:58.023609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:28:58.050962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 431/860] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:28:57 2022] POST /identity/v3/projects => generated 311 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:29:00.329731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19e57a3-58c1-41cd-ba3c-629f2e37640f None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:00.330272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19e57a3-58c1-41cd-ba3c-629f2e37640f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:00.330272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19e57a3-58c1-41cd-ba3c-629f2e37640f None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:00.331644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 430/861] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:29:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:29:00.338154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fbdfeb-8970-4583-b8cb-668da84a03d5 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:00.338760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fbdfeb-8970-4583-b8cb-668da84a03d5 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:00.338890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fbdfeb-8970-4583-b8cb-668da84a03d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:00.358503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-89fbdfeb-8970-4583-b8cb-668da84a03d5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:29:00.359718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-89fbdfeb-8970-4583-b8cb-668da84a03d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:29:00.431356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 432/862] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:29:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:29:00.699815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19e57a3-58c1-41cd-ba3c-629f2e37640f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:00.812216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19e57a3-58c1-41cd-ba3c-629f2e37640f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:00.849003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6180291-f329-4f69-9574-32e137bfe2e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:00.849655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6180291-f329-4f69-9574-32e137bfe2e5 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:00.849939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6180291-f329-4f69-9574-32e137bfe2e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:00.850224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6180291-f329-4f69-9574-32e137bfe2e5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:00.851502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6180291-f329-4f69-9574-32e137bfe2e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:00.852951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6180291-f329-4f69-9574-32e137bfe2e5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:00.854545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 431/863] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:29:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:00.862253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fbdfeb-8970-4583-b8cb-668da84a03d5 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:00.966987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fbdfeb-8970-4583-b8cb-668da84a03d5 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:00.984384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:01.004966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b9ee85-c140-43f7-b23e-1d1f11ac603d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:01.005581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b9ee85-c140-43f7-b23e-1d1f11ac603d None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:01.005861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b9ee85-c140-43f7-b23e-1d1f11ac603d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:01.006133 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b9ee85-c140-43f7-b23e-1d1f11ac603d None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:01.006825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b9ee85-c140-43f7-b23e-1d1f11ac603d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:01.008419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b9ee85-c140-43f7-b23e-1d1f11ac603d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:01.035389 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 433/864] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:29:00 2022] POST /identity/v3/users => generated 286 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:29:03.296332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a971b-59cd-4698-9d83-74f77cd157e8 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:03.296891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a971b-59cd-4698-9d83-74f77cd157e8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:03.296891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a971b-59cd-4698-9d83-74f77cd157e8 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:03.298250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 432/865] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:29:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:29:03.304153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ab3c8-90d1-4ed4-90a3-f94477ccf409 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:03.304666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ab3c8-90d1-4ed4-90a3-f94477ccf409 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:03.304813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ab3c8-90d1-4ed4-90a3-f94477ccf409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:03.325023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d94ab3c8-90d1-4ed4-90a3-f94477ccf409 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:29:03.326257 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d94ab3c8-90d1-4ed4-90a3-f94477ccf409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:29:03.402338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 434/866] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:29:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:29:03.680124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0a971b-59cd-4698-9d83-74f77cd157e8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:03.758741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:03.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0a971b-59cd-4698-9d83-74f77cd157e8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:03.842337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feac55d1-69c2-4c3f-a79f-c8bdeeb5f863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:03.843064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac55d1-69c2-4c3f-a79f-c8bdeeb5f863 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:03.843344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac55d1-69c2-4c3f-a79f-c8bdeeb5f863 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:03.843617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac55d1-69c2-4c3f-a79f-c8bdeeb5f863 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:03.855077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feac55d1-69c2-4c3f-a79f-c8bdeeb5f863 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-feac55d1-69c2-4c3f-a79f-c8bdeeb5f863 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:29:03.860193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:29:03.861926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:29:03.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 433/867] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:29:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:29:03.869288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94ab3c8-90d1-4ed4-90a3-f94477ccf409 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:03.991817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94ab3c8-90d1-4ed4-90a3-f94477ccf409 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:03.993522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:04.030766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:04.031397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:04.031666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:04.031937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:04.032612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:29:04.032912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:04.034306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:04.044392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 435/868] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:29:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:04.053323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feac55d1-69c2-4c3f-a79f-c8bdeeb5f863 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:04.161407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feac55d1-69c2-4c3f-a79f-c8bdeeb5f863 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:04.190357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:04.200751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724d7a80-d808-4059-b26f-13d6d0779a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:04.201399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724d7a80-d808-4059-b26f-13d6d0779a90 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:04.201679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724d7a80-d808-4059-b26f-13d6d0779a90 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:04.201954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724d7a80-d808-4059-b26f-13d6d0779a90 None admin] PATH_INFO: `/v3/users/94bd7fb24aaf4e3a85947ee657bb02bb` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:04.218715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724d7a80-d808-4059-b26f-13d6d0779a90 None admin] RBAC: Authorizing `identity:get_user(user_id=94bd7fb24aaf4e3a85947ee657bb02bb)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:04.220296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724d7a80-d808-4059-b26f-13d6d0779a90 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:04.221988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 434/869] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:29:04 2022] GET /identity/v3/users/94bd7fb24aaf4e3a85947ee657bb02bb => generated 286 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:04.228636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:04.337352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3b9c67-0490-47d6-a17d-62c2aacc78a5 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:04.376536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e406af4-e339-49c6-bc43-325a537f4bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:04.377295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e406af4-e339-49c6-bc43-325a537f4bb8 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:04.377640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e406af4-e339-49c6-bc43-325a537f4bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:04.377964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e406af4-e339-49c6-bc43-325a537f4bb8 None admin] PATH_INFO: `/v3/projects/fac4d51867bc4ffbad1b4ae51e549182` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:04.379247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e406af4-e339-49c6-bc43-325a537f4bb8 None admin] RBAC: Authorizing `identity:get_project(project_id=fac4d51867bc4ffbad1b4ae51e549182)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:04.380801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e406af4-e339-49c6-bc43-325a537f4bb8 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:04.382564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 436/870] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:29:04 2022] GET /identity/v3/projects/fac4d51867bc4ffbad1b4ae51e549182 => generated 311 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:29:04.389431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724d7a80-d808-4059-b26f-13d6d0779a90 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:04.498804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724d7a80-d808-4059-b26f-13d6d0779a90 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:04.535145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cfc7e7-063b-4246-97ca-1b687d7ee1b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:04.535818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cfc7e7-063b-4246-97ca-1b687d7ee1b5 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:04.536102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cfc7e7-063b-4246-97ca-1b687d7ee1b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:04.536379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cfc7e7-063b-4246-97ca-1b687d7ee1b5 None admin] PATH_INFO: `/v3/projects/fac4d51867bc4ffbad1b4ae51e549182/users/94bd7fb24aaf4e3a85947ee657bb02bb/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:04.538340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cfc7e7-063b-4246-97ca-1b687d7ee1b5 None admin] RBAC: Authorizing `identity:create_grant(project_id=fac4d51867bc4ffbad1b4ae51e549182, user_id=94bd7fb24aaf4e3a85947ee657bb02bb, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:04.539828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cfc7e7-063b-4246-97ca-1b687d7ee1b5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:04.556921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 435/871] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:29:04 2022] PUT /identity/v3/projects/fac4d51867bc4ffbad1b4ae51e549182/users/94bd7fb24aaf4e3a85947ee657bb02bb/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:29:06.809629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f728b44-2d7a-455d-92e8-a5514f033261 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:06.810198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f728b44-2d7a-455d-92e8-a5514f033261 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:06.810198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f728b44-2d7a-455d-92e8-a5514f033261 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:06.811529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 437/872] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:29:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:29:06.817701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ce774-9163-4c69-b512-bc5fca409295 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:06.818573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ce774-9163-4c69-b512-bc5fca409295 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:06.818573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ce774-9163-4c69-b512-bc5fca409295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:06.838539 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5a0ce774-9163-4c69-b512-bc5fca409295 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:29:06.840476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5a0ce774-9163-4c69-b512-bc5fca409295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:29:06.884269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:06.913925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 436/873] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:29:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:29:07.182582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f728b44-2d7a-455d-92e8-a5514f033261 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:07.219727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:07.305556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f728b44-2d7a-455d-92e8-a5514f033261 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:07.346473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a9e248-75d9-4f3c-8a22-dabd31816cfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:07.347150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9e248-75d9-4f3c-8a22-dabd31816cfd None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:07.347427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9e248-75d9-4f3c-8a22-dabd31816cfd None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:07.347958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9e248-75d9-4f3c-8a22-dabd31816cfd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:07.361299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a9e248-75d9-4f3c-8a22-dabd31816cfd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-16a9e248-75d9-4f3c-8a22-dabd31816cfd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:29:07.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:29:07.368205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:29:07.369720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 438/874] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:29:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:29:07.377392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0ce774-9163-4c69-b512-bc5fca409295 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:07.497346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0ce774-9163-4c69-b512-bc5fca409295 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:07.536603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:07.537235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:07.537517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:07.537802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:07.538524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:29:07.538818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:07.540244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:07.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 437/875] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:29:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:07.558189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a9e248-75d9-4f3c-8a22-dabd31816cfd None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:07.721900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:07.737374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a9e248-75d9-4f3c-8a22-dabd31816cfd None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:07.796401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224e6a54-2673-4504-9e8d-f191d0c84c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:07.797383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224e6a54-2673-4504-9e8d-f191d0c84c46 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:07.797862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224e6a54-2673-4504-9e8d-f191d0c84c46 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:07.798319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224e6a54-2673-4504-9e8d-f191d0c84c46 None admin] PATH_INFO: `/v3/users/94bd7fb24aaf4e3a85947ee657bb02bb` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:07.800463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224e6a54-2673-4504-9e8d-f191d0c84c46 None admin] RBAC: Authorizing `identity:get_user(user_id=94bd7fb24aaf4e3a85947ee657bb02bb)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:07.803232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224e6a54-2673-4504-9e8d-f191d0c84c46 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:07.806302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 439/876] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:29:07 2022] GET /identity/v3/users/94bd7fb24aaf4e3a85947ee657bb02bb => generated 286 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:07.813067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:07.852779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:07.927749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf00e51-7477-4c5d-a6ea-52ee779bb1d3 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:07.965341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438eefc4-1c46-4736-a5f7-517f8ccc3016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:07.965964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438eefc4-1c46-4736-a5f7-517f8ccc3016 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:07.966241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438eefc4-1c46-4736-a5f7-517f8ccc3016 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:07.966543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438eefc4-1c46-4736-a5f7-517f8ccc3016 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:08.043155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438eefc4-1c46-4736-a5f7-517f8ccc3016 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:08.044586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438eefc4-1c46-4736-a5f7-517f8ccc3016 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:08.079600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:08.153601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 438/877] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:29:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 00:29:08.159713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224e6a54-2673-4504-9e8d-f191d0c84c46 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:08.283767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224e6a54-2673-4504-9e8d-f191d0c84c46 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:08.339694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ded8f5-6e0b-47c7-8cc2-b30c106e46ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:08.340652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ded8f5-6e0b-47c7-8cc2-b30c106e46ae None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:08.341124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ded8f5-6e0b-47c7-8cc2-b30c106e46ae None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:08.341594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ded8f5-6e0b-47c7-8cc2-b30c106e46ae None admin] PATH_INFO: `/v3/projects/fac4d51867bc4ffbad1b4ae51e549182` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:08.343541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ded8f5-6e0b-47c7-8cc2-b30c106e46ae None admin] RBAC: Authorizing `identity:get_project(project_id=fac4d51867bc4ffbad1b4ae51e549182)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:08.346151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ded8f5-6e0b-47c7-8cc2-b30c106e46ae None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:08.349396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 440/878] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:29:08 2022] GET /identity/v3/projects/fac4d51867bc4ffbad1b4ae51e549182 => generated 311 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:08.356287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438eefc4-1c46-4736-a5f7-517f8ccc3016 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:08.457301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438eefc4-1c46-4736-a5f7-517f8ccc3016 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:08.508592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dc4ef3-e023-4d08-a1f2-dbf97c5adc9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:08.509634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc4ef3-e023-4d08-a1f2-dbf97c5adc9f None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:08.510109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc4ef3-e023-4d08-a1f2-dbf97c5adc9f None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:08.510613 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc4ef3-e023-4d08-a1f2-dbf97c5adc9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:08.512026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dc4ef3-e023-4d08-a1f2-dbf97c5adc9f None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=fac4d51867bc4ffbad1b4ae51e549182, user.id=94bd7fb24aaf4e3a85947ee657bb02bb) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:29:08.512560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dc4ef3-e023-4d08-a1f2-dbf97c5adc9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:08.515255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dc4ef3-e023-4d08-a1f2-dbf97c5adc9f None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:08.520545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:08.527145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 439/879] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:29:08 2022] GET /identity/v3/role_assignments?user.id=94bd7fb24aaf4e3a85947ee657bb02bb&scope.project.id=fac4d51867bc4ffbad1b4ae51e549182&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:10.812016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894dfea-0d60-415e-8d50-606512ad7110 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:10.812549 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894dfea-0d60-415e-8d50-606512ad7110 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:10.812549 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894dfea-0d60-415e-8d50-606512ad7110 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:10.813769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 441/880] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:29:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:29:10.819477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6501acec-eb47-4651-8d82-3e4ab16e5a30 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:10.819963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6501acec-eb47-4651-8d82-3e4ab16e5a30 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:10.820056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6501acec-eb47-4651-8d82-3e4ab16e5a30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:10.838756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-6501acec-eb47-4651-8d82-3e4ab16e5a30 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:29:10.840713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6501acec-eb47-4651-8d82-3e4ab16e5a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:29:10.910815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 440/881] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:29:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:29:11.184708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6894dfea-0d60-415e-8d50-606512ad7110 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:11.217476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:11.343648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6894dfea-0d60-415e-8d50-606512ad7110 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:11.393812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1f67b3-54fa-4b41-9dc2-d37266d041e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:11.394518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f67b3-54fa-4b41-9dc2-d37266d041e7 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:11.394853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f67b3-54fa-4b41-9dc2-d37266d041e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:11.395180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1f67b3-54fa-4b41-9dc2-d37266d041e7 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:11.395915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1f67b3-54fa-4b41-9dc2-d37266d041e7 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:11.397343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1f67b3-54fa-4b41-9dc2-d37266d041e7 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:11.415196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 442/882] 10.222.0.26 () {60 vars in 1138 bytes} [Wed Oct 19 00:29:11 2022] POST /identity/v3/domains => generated 267 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:29:13.698428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be355977-5ee5-444a-ba2d-d16cde877c0f None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:13.699015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be355977-5ee5-444a-ba2d-d16cde877c0f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:13.699111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be355977-5ee5-444a-ba2d-d16cde877c0f None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:13.700499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 441/883] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:29:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:29:13.706370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a36077-c30f-4961-827d-b18e0d6f5cad None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:13.706955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a36077-c30f-4961-827d-b18e0d6f5cad None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:13.706955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a36077-c30f-4961-827d-b18e0d6f5cad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:13.726634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a2a36077-c30f-4961-827d-b18e0d6f5cad None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:29:13.728024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a2a36077-c30f-4961-827d-b18e0d6f5cad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:29:13.800043 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 443/884] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:29:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:29:14.069731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be355977-5ee5-444a-ba2d-d16cde877c0f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:14.187010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be355977-5ee5-444a-ba2d-d16cde877c0f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:14.223827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:14.232142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fbde90-f877-40a0-a0dd-c25e7816544d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:14.232879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fbde90-f877-40a0-a0dd-c25e7816544d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:14.233201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fbde90-f877-40a0-a0dd-c25e7816544d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:14.233522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fbde90-f877-40a0-a0dd-c25e7816544d None admin] PATH_INFO: `/v3/domains/3ee2109b14124facb5572acf24b2fbec` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:14.246774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fbde90-f877-40a0-a0dd-c25e7816544d None admin] RBAC: Authorizing `identity:get_domain(domain_id=3ee2109b14124facb5572acf24b2fbec)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:14.248263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fbde90-f877-40a0-a0dd-c25e7816544d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:14.250007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 442/885] 10.222.0.26 () {56 vars in 1183 bytes} [Wed Oct 19 00:29:14 2022] GET /identity/v3/domains/3ee2109b14124facb5572acf24b2fbec => generated 267 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:29:14.256873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a36077-c30f-4961-827d-b18e0d6f5cad None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:14.280112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:14.369561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a36077-c30f-4961-827d-b18e0d6f5cad None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:14.407768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d9ee80-8fdb-4525-9f15-8e07b416c012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:14.408378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9ee80-8fdb-4525-9f15-8e07b416c012 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:14.408658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9ee80-8fdb-4525-9f15-8e07b416c012 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:14.408928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9ee80-8fdb-4525-9f15-8e07b416c012 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:14.409585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d9ee80-8fdb-4525-9f15-8e07b416c012 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:14.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d9ee80-8fdb-4525-9f15-8e07b416c012 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:14.441626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 444/886] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:29:14 2022] POST /identity/v3/projects => generated 361 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:29:16.741904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08acc4e2-bf75-415b-bf57-845fe421245f None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:16.742569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08acc4e2-bf75-415b-bf57-845fe421245f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:16.742569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08acc4e2-bf75-415b-bf57-845fe421245f None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:16.744047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 443/887] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:29:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:29:16.750632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db1cb7-71bb-4f21-975d-a08ee32d3cbb None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:16.751191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db1cb7-71bb-4f21-975d-a08ee32d3cbb None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:16.751191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db1cb7-71bb-4f21-975d-a08ee32d3cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:16.772171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-31db1cb7-71bb-4f21-975d-a08ee32d3cbb None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:29:16.773480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-31db1cb7-71bb-4f21-975d-a08ee32d3cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:29:16.843933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:16.849751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 445/888] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:29:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:29:17.120388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08acc4e2-bf75-415b-bf57-845fe421245f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:17.230983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08acc4e2-bf75-415b-bf57-845fe421245f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:17.269882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aafa786-11bb-4a12-9948-0a410286e7d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:17.270631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aafa786-11bb-4a12-9948-0a410286e7d8 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:17.271009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aafa786-11bb-4a12-9948-0a410286e7d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:17.271328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aafa786-11bb-4a12-9948-0a410286e7d8 None admin] PATH_INFO: `/v3/domains/3ee2109b14124facb5572acf24b2fbec` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:17.272715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aafa786-11bb-4a12-9948-0a410286e7d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=3ee2109b14124facb5572acf24b2fbec)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:17.274270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aafa786-11bb-4a12-9948-0a410286e7d8 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:17.276113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 444/889] 10.222.0.26 () {56 vars in 1183 bytes} [Wed Oct 19 00:29:17 2022] GET /identity/v3/domains/3ee2109b14124facb5572acf24b2fbec => generated 267 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:17.282633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db1cb7-71bb-4f21-975d-a08ee32d3cbb None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:17.393085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db1cb7-71bb-4f21-975d-a08ee32d3cbb None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:17.429239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc3cdb5-d0f3-4b8f-ad73-5f437bedec9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:17.429917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc3cdb5-d0f3-4b8f-ad73-5f437bedec9d None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:17.430242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc3cdb5-d0f3-4b8f-ad73-5f437bedec9d None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:17.430673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc3cdb5-d0f3-4b8f-ad73-5f437bedec9d None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:17.431390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc3cdb5-d0f3-4b8f-ad73-5f437bedec9d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:17.432955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc3cdb5-d0f3-4b8f-ad73-5f437bedec9d None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:17.455659 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 446/890] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:29:17 2022] POST /identity/v3/users => generated 311 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:29:19.767098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5aabde8-4c0d-4713-863c-80c83df05123 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:19.767651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5aabde8-4c0d-4713-863c-80c83df05123 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:19.767651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5aabde8-4c0d-4713-863c-80c83df05123 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:19.768984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 445/891] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:29:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:29:19.774833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145763c2-926c-450d-918a-c01968420982 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:19.775365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145763c2-926c-450d-918a-c01968420982 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:19.775365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145763c2-926c-450d-918a-c01968420982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:19.796563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-145763c2-926c-450d-918a-c01968420982 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:29:19.797778 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-145763c2-926c-450d-918a-c01968420982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:29:19.861977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:19.890242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 447/892] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:29:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:29:20.163876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5aabde8-4c0d-4713-863c-80c83df05123 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:20.230032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:20.276035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5aabde8-4c0d-4713-863c-80c83df05123 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:20.312742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aff62f3-ac82-4e51-92e0-eaec8bb17fbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:20.313382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aff62f3-ac82-4e51-92e0-eaec8bb17fbe None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:20.313674 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aff62f3-ac82-4e51-92e0-eaec8bb17fbe None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:20.313944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aff62f3-ac82-4e51-92e0-eaec8bb17fbe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:20.325460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aff62f3-ac82-4e51-92e0-eaec8bb17fbe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-0aff62f3-ac82-4e51-92e0-eaec8bb17fbe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:29:20.330379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:29:20.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:29:20.332698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 446/893] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:29:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:29:20.341340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145763c2-926c-450d-918a-c01968420982 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:20.451683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145763c2-926c-450d-918a-c01968420982 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:20.489562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:20.490176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:20.490453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:20.490750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:20.491428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:29:20.491730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:20.493116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:20.502701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 448/894] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:29:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:29:20.512450 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aff62f3-ac82-4e51-92e0-eaec8bb17fbe None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:20.639806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aff62f3-ac82-4e51-92e0-eaec8bb17fbe None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:20.659232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:20.678854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d6cb14-b914-4a29-a861-ee5d219a0a30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:20.679497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d6cb14-b914-4a29-a861-ee5d219a0a30 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:20.679776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d6cb14-b914-4a29-a861-ee5d219a0a30 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:20.680050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d6cb14-b914-4a29-a861-ee5d219a0a30 None admin] PATH_INFO: `/v3/users/cf512e52a03a4d8a98e3a0b0bae104f0` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:20.695713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d6cb14-b914-4a29-a861-ee5d219a0a30 None admin] RBAC: Authorizing `identity:get_user(user_id=cf512e52a03a4d8a98e3a0b0bae104f0)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:20.697253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d6cb14-b914-4a29-a861-ee5d219a0a30 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:20.699096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 447/895] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:29:20 2022] GET /identity/v3/users/cf512e52a03a4d8a98e3a0b0bae104f0 => generated 311 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:20.708348 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:20.764427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:20.830142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5018abb-912c-476b-875e-12bc27d2fba0 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:20.872017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093d0cd9-4f3e-4b26-bf47-2e659bf0e719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:20.872754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093d0cd9-4f3e-4b26-bf47-2e659bf0e719 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:20.873078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093d0cd9-4f3e-4b26-bf47-2e659bf0e719 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:20.873395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093d0cd9-4f3e-4b26-bf47-2e659bf0e719 None admin] PATH_INFO: `/v3/projects/f9b5686bfa874e159dbc040cc6eaa7fe` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:20.874761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093d0cd9-4f3e-4b26-bf47-2e659bf0e719 None admin] RBAC: Authorizing `identity:get_project(project_id=f9b5686bfa874e159dbc040cc6eaa7fe)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:20.876318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093d0cd9-4f3e-4b26-bf47-2e659bf0e719 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:20.878146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 449/896] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:29:20 2022] GET /identity/v3/projects/f9b5686bfa874e159dbc040cc6eaa7fe => generated 361 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:20.884527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d6cb14-b914-4a29-a861-ee5d219a0a30 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:21.001323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d6cb14-b914-4a29-a861-ee5d219a0a30 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:21.040385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1813e342-48c0-4ab0-9b05-5e726cd5ad76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:21.041070 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813e342-48c0-4ab0-9b05-5e726cd5ad76 None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:21.041348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813e342-48c0-4ab0-9b05-5e726cd5ad76 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:21.041622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813e342-48c0-4ab0-9b05-5e726cd5ad76 None admin] PATH_INFO: `/v3/projects/f9b5686bfa874e159dbc040cc6eaa7fe/users/cf512e52a03a4d8a98e3a0b0bae104f0/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:21.044259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1813e342-48c0-4ab0-9b05-5e726cd5ad76 None admin] RBAC: Authorizing `identity:create_grant(project_id=f9b5686bfa874e159dbc040cc6eaa7fe, user_id=cf512e52a03a4d8a98e3a0b0bae104f0, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:21.045725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1813e342-48c0-4ab0-9b05-5e726cd5ad76 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:21.060521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 448/897] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:29:20 2022] PUT /identity/v3/projects/f9b5686bfa874e159dbc040cc6eaa7fe/users/cf512e52a03a4d8a98e3a0b0bae104f0/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:29:23.365865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a2d8b3-872f-4b96-a48b-a872fd4269c8 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:23.366461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a2d8b3-872f-4b96-a48b-a872fd4269c8 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:23.366461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a2d8b3-872f-4b96-a48b-a872fd4269c8 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:23.367766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 450/898] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:29:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:29:23.374841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd72df3-c5c2-4a73-986e-50ed3e3056d4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:23.375304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd72df3-c5c2-4a73-986e-50ed3e3056d4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:23.375419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd72df3-c5c2-4a73-986e-50ed3e3056d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:23.395202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-3cd72df3-c5c2-4a73-986e-50ed3e3056d4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:29:23.396419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3cd72df3-c5c2-4a73-986e-50ed3e3056d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:29:23.436880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:23.482648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 449/899] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:29:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:29:23.755218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a2d8b3-872f-4b96-a48b-a872fd4269c8 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:23.866930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a2d8b3-872f-4b96-a48b-a872fd4269c8 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:23.886087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:23.907144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78ac26c-7d31-4e59-852a-92619408c47b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:23.907778 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ac26c-7d31-4e59-852a-92619408c47b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:23.908057 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ac26c-7d31-4e59-852a-92619408c47b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:23.908330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ac26c-7d31-4e59-852a-92619408c47b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:23.920502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78ac26c-7d31-4e59-852a-92619408c47b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-e78ac26c-7d31-4e59-852a-92619408c47b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:29:23.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:29:23.927139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:29:23.928802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 451/900] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:29:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:29:23.936607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd72df3-c5c2-4a73-986e-50ed3e3056d4 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:24.098881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd72df3-c5c2-4a73-986e-50ed3e3056d4 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:24.163432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:24.164481 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:24.164970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:24.165453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:24.167280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:29:24.167781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:24.170066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:24.186195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 450/901] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:29:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:24.193264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78ac26c-7d31-4e59-852a-92619408c47b None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:24.324983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78ac26c-7d31-4e59-852a-92619408c47b None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:24.365438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807318ec-6009-44ef-94c1-61516413642e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:24.366063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807318ec-6009-44ef-94c1-61516413642e None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:24.366728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807318ec-6009-44ef-94c1-61516413642e None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:24.366728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807318ec-6009-44ef-94c1-61516413642e None admin] PATH_INFO: `/v3/users/cf512e52a03a4d8a98e3a0b0bae104f0` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:24.368051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807318ec-6009-44ef-94c1-61516413642e None admin] RBAC: Authorizing `identity:get_user(user_id=cf512e52a03a4d8a98e3a0b0bae104f0)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:24.369581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807318ec-6009-44ef-94c1-61516413642e None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:24.371536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 452/902] 10.222.0.26 () {56 vars in 1177 bytes} [Wed Oct 19 00:29:24 2022] GET /identity/v3/users/cf512e52a03a4d8a98e3a0b0bae104f0 => generated 311 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:24.377798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:24.426743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:24.552307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f0931e-94ff-4043-9990-71679fb13254 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:24.608798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50190926-2034-498a-90b8-0d288f0044b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:24.609410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50190926-2034-498a-90b8-0d288f0044b5 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:24.609691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50190926-2034-498a-90b8-0d288f0044b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:24.609967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50190926-2034-498a-90b8-0d288f0044b5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:24.680944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50190926-2034-498a-90b8-0d288f0044b5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:24.682944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50190926-2034-498a-90b8-0d288f0044b5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:24.711566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:24.794344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 451/903] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:29:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 00:29:24.801042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807318ec-6009-44ef-94c1-61516413642e None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:24.851326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:24.911807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807318ec-6009-44ef-94c1-61516413642e None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:24.948520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710dc157-7b7f-430e-ae32-e3d1a7ef9758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:24.949245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710dc157-7b7f-430e-ae32-e3d1a7ef9758 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:24.949521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710dc157-7b7f-430e-ae32-e3d1a7ef9758 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:24.949785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710dc157-7b7f-430e-ae32-e3d1a7ef9758 None admin] PATH_INFO: `/v3/projects/f9b5686bfa874e159dbc040cc6eaa7fe` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:24.950998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710dc157-7b7f-430e-ae32-e3d1a7ef9758 None admin] RBAC: Authorizing `identity:get_project(project_id=f9b5686bfa874e159dbc040cc6eaa7fe)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:24.952567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710dc157-7b7f-430e-ae32-e3d1a7ef9758 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:24.954246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 453/904] 10.222.0.26 () {56 vars in 1186 bytes} [Wed Oct 19 00:29:24 2022] GET /identity/v3/projects/f9b5686bfa874e159dbc040cc6eaa7fe => generated 361 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:24.960630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50190926-2034-498a-90b8-0d288f0044b5 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:29:25.075421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50190926-2034-498a-90b8-0d288f0044b5 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:29:25.108946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:25.117740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7779bddc-05cf-4697-ab16-3263533c87f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:29:25.118368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7779bddc-05cf-4697-ab16-3263533c87f2 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:29:25.118687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7779bddc-05cf-4697-ab16-3263533c87f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:29:25.118958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7779bddc-05cf-4697-ab16-3263533c87f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:29:25.119763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7779bddc-05cf-4697-ab16-3263533c87f2 None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=f9b5686bfa874e159dbc040cc6eaa7fe, user.id=cf512e52a03a4d8a98e3a0b0bae104f0) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:29:25.120277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7779bddc-05cf-4697-ab16-3263533c87f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:29:25.121744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7779bddc-05cf-4697-ab16-3263533c87f2 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:29:25.128608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 452/905] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:29:24 2022] GET /identity/v3/role_assignments?user.id=cf512e52a03a4d8a98e3a0b0bae104f0&scope.project.id=f9b5686bfa874e159dbc040cc6eaa7fe&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:29:34.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=44 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:29:35.125767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:23.576811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee80b26-1f1a-4d1b-9ef5-a07a2dfff050 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:23.577737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee80b26-1f1a-4d1b-9ef5-a07a2dfff050 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:23.577737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee80b26-1f1a-4d1b-9ef5-a07a2dfff050 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:23.579004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 454/906] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:30:23.585602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeb2435-0054-4d2d-a963-49f30568c9f5 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:23.586519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeb2435-0054-4d2d-a963-49f30568c9f5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:23.586519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeb2435-0054-4d2d-a963-49f30568c9f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:23.612771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-4eeb2435-0054-4d2d-a963-49f30568c9f5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:23.614152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4eeb2435-0054-4d2d-a963-49f30568c9f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:23.702937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 453/907] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:30:24.026011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee80b26-1f1a-4d1b-9ef5-a07a2dfff050 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:24.173592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee80b26-1f1a-4d1b-9ef5-a07a2dfff050 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:24.214360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:24.215291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:24.215852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:24.218693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:24.233449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:24.235400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:24.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:24.259905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:24.260917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:24.260917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:24.260917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:24.266991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 455/908] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:24.267598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeb2435-0054-4d2d-a963-49f30568c9f5 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:24.405951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeb2435-0054-4d2d-a963-49f30568c9f5 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:24.436111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:24.443941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb6260-98ea-49ac-8ce8-a47cd32719b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:24.444788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb6260-98ea-49ac-8ce8-a47cd32719b1 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:24.445233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb6260-98ea-49ac-8ce8-a47cd32719b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:24.445914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb6260-98ea-49ac-8ce8-a47cd32719b1 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:24.447130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cb6260-98ea-49ac-8ce8-a47cd32719b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:24.447422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cb6260-98ea-49ac-8ce8-a47cd32719b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:24.449095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cb6260-98ea-49ac-8ce8-a47cd32719b1 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:24.471216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 454/909] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:24.473294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:24.540396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:24.619628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4446c8-6270-4f87-8ac7-d5d8c732ba48 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:24.666800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912bc362-4fd4-4f54-b361-637c6d48ac31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:24.667183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912bc362-4fd4-4f54-b361-637c6d48ac31 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:24.667464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912bc362-4fd4-4f54-b361-637c6d48ac31 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:24.667738 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912bc362-4fd4-4f54-b361-637c6d48ac31 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:24.668594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912bc362-4fd4-4f54-b361-637c6d48ac31 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:24.670678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912bc362-4fd4-4f54-b361-637c6d48ac31 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:24.699464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 456/910] 10.222.0.26 () {60 vars in 1132 bytes} [Wed Oct 19 00:30:24 2022] POST /identity/v3/users => generated 251 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:30:27.241454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccefe9af-b52a-418e-8d19-8c9c180af899 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:27.241454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccefe9af-b52a-418e-8d19-8c9c180af899 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:27.241454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccefe9af-b52a-418e-8d19-8c9c180af899 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:27.244372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 455/911] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:30:27.251827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a225bfba-2163-4935-8523-ca72b03c197f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:27.252468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a225bfba-2163-4935-8523-ca72b03c197f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:27.252565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a225bfba-2163-4935-8523-ca72b03c197f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:27.280396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a225bfba-2163-4935-8523-ca72b03c197f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:27.284572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a225bfba-2163-4935-8523-ca72b03c197f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:27.386434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 457/912] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:30:27.705905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccefe9af-b52a-418e-8d19-8c9c180af899 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:27.864135 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccefe9af-b52a-418e-8d19-8c9c180af899 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:27.924242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a429816a-4ffa-4969-8e79-adecc1285fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:27.925504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a429816a-4ffa-4969-8e79-adecc1285fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:27.925792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a429816a-4ffa-4969-8e79-adecc1285fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:27.926067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a429816a-4ffa-4969-8e79-adecc1285fb9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:27.942897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a429816a-4ffa-4969-8e79-adecc1285fb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-a429816a-4ffa-4969-8e79-adecc1285fb9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:30:27.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:27.957085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:27.958128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 456/913] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:30:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:27.958958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a225bfba-2163-4935-8523-ca72b03c197f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:28.038296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:28.134185 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a225bfba-2163-4935-8523-ca72b03c197f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:28.190474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:28.191166 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:28.191422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:28.191675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:28.192399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] RBAC: Adding query filter params (name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:28.192817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:28.197438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:28.205405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 458/914] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:30:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:28.211387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a429816a-4ffa-4969-8e79-adecc1285fb9 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:28.289360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:28.360967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a429816a-4ffa-4969-8e79-adecc1285fb9 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:28.403856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:28.404410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:28.404751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:28.405088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:28.418750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:28.420301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:28.438523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:28.439598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:28.440525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:28.440525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:28.440525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:28.440525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 457/915] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:28.447285 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:28.588364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:28.592029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0724c4f-9e87-4b72-a9a8-5c9318eb4e98 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:28.639529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:28.640663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:28.640872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:28.641191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:28.641971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:28.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:28.643863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:28.663678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 459/916] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:28.665591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:28.804656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f263d4-7f5f-43cf-b6e5-25108be82a80 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:28.807388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:28.849788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:28.850862 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:28.851358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:28.851868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:28.861119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:28.863636 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:30:28.878724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:28.879991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 00:30:28.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:28.884610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 458/917] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:30:28.890557 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:29.021265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a623561a-2836-40d0-9cf6-89b884b1d1d2 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:29.067866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:29.068926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:29.069460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:29.069963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:29.071923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:29.072462 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:29.074674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:29.101754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 460/918] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Oct 19 00:30:28 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:29.113135 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:29.249579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232df3b9-0ca2-4d0f-937b-69287e877a0d None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:29.298361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:29.299664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:29.300373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:29.300836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:29.317808 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:29.319604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:29.338406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:29.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:29.345115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:29.345115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:29.345115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:29.345115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 459/919] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:29.345959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:29.377751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:29.518932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d2b9bb-f8c3-4429-9f91-16e1ef518b65 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:29.572487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:29.573633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:29.574252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:29.575099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:29.576302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:29.576852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:29.579381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:29.613697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 461/920] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:29.615627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:29.651269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:29.763992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa779d3-4b35-4c81-97df-15dbe8278f76 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:29.810990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:29.811658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:29.811930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:29.812295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:29.821139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:29.823299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:30:29.837851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:29.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:29.840036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 460/921] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:30:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:29.845242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:29.956957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:29.992369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061843cf-e6b1-4e04-94a7-a9f081a6f2ca None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:30.052401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3139d1dd-1710-49c7-9d23-70ab66645c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:30.053523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3139d1dd-1710-49c7-9d23-70ab66645c54 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:30.053969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3139d1dd-1710-49c7-9d23-70ab66645c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:30.057896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3139d1dd-1710-49c7-9d23-70ab66645c54 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:30.057896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3139d1dd-1710-49c7-9d23-70ab66645c54 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:30.057896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3139d1dd-1710-49c7-9d23-70ab66645c54 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:30.059669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3139d1dd-1710-49c7-9d23-70ab66645c54 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:30.083182 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 462/922] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:30:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:30.091258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:30.198805 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:30.236183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0568393-3a43-41ec-881d-69b4481ad1ae None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:30.293015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c799f08-e709-4ce9-bd52-711d7e1e9f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:30.293848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c799f08-e709-4ce9-bd52-711d7e1e9f3d None admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:30.294132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c799f08-e709-4ce9-bd52-711d7e1e9f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:30.294412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c799f08-e709-4ce9-bd52-711d7e1e9f3d None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/93d2a83a28fd4608994317c2c7b6a723/roles/8b6c794a10cf4fcdb33edf34796d4819` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:30.313996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c799f08-e709-4ce9-bd52-711d7e1e9f3d None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=93d2a83a28fd4608994317c2c7b6a723, role_id=8b6c794a10cf4fcdb33edf34796d4819)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:30.315589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c799f08-e709-4ce9-bd52-711d7e1e9f3d None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:30.332499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 461/923] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:30:30 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/93d2a83a28fd4608994317c2c7b6a723/roles/8b6c794a10cf4fcdb33edf34796d4819 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:30:32.738529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07941b-f111-44d2-a466-12e1a8a499fe None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:32.738529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07941b-f111-44d2-a466-12e1a8a499fe None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:32.738529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07941b-f111-44d2-a466-12e1a8a499fe None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:32.739457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 463/924] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:30:32.746272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362fe1a8-9748-4a30-bdc5-fc1f6f876a2b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:32.746790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362fe1a8-9748-4a30-bdc5-fc1f6f876a2b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:32.746944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362fe1a8-9748-4a30-bdc5-fc1f6f876a2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:32.766715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-362fe1a8-9748-4a30-bdc5-fc1f6f876a2b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:32.767989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-362fe1a8-9748-4a30-bdc5-fc1f6f876a2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:32.848289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 462/925] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:30:33.132795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e07941b-f111-44d2-a466-12e1a8a499fe None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:33.271540 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e07941b-f111-44d2-a466-12e1a8a499fe None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:33.316915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b392db-d541-44b5-b3e7-018af7f5db5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:33.317559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b392db-d541-44b5-b3e7-018af7f5db5c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:33.317841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b392db-d541-44b5-b3e7-018af7f5db5c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:33.318109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b392db-d541-44b5-b3e7-018af7f5db5c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:33.326050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:33.330396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b392db-d541-44b5-b3e7-018af7f5db5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-56b392db-d541-44b5-b3e7-018af7f5db5c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:30:33.336397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:33.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:33.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 464/926] 10.222.0.26 () {56 vars in 1102 bytes} [Wed Oct 19 00:30:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:33.347227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362fe1a8-9748-4a30-bdc5-fc1f6f876a2b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:33.413403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:33.467559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362fe1a8-9748-4a30-bdc5-fc1f6f876a2b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:33.513841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:33.514662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:33.515071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:33.515566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:33.516428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:33.516798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:33.518338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:33.528832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 463/927] 10.222.0.26 () {56 vars in 1103 bytes} [Wed Oct 19 00:30:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:33.535294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b392db-d541-44b5-b3e7-018af7f5db5c None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:33.677223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b392db-d541-44b5-b3e7-018af7f5db5c None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:33.721901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:33.722857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:33.723126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:33.723404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:33.736037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:33.737914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:33.757320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:33.764379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:33.765481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:33.765481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:33.765481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:33.765481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 465/928] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:30:33.765767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:33.897879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d9217f-36b8-4e2a-ad13-08136c5959a7 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:33.924129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:33.949110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:33.949884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:33.950226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:33.950541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:33.951220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:33.951521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:33.952906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:33.967896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 464/929] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:33.974899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:34.041766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:34.109486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82a16ee-4905-4e70-a03b-fe86838a3fc8 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:34.151036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:34.151829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:34.152128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:34.152399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:34.162794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:34.164703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:30:34.181428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:34.182631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:34.183847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 466/930] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:33 2022] GET /identity/v3/users/placement => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:34.188708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:34.356669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb36823b-1d82-4983-abaa-0eeaaf8ae13c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:34.406612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:34.407367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:34.407708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:34.408050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:34.408863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:34.409234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:34.410787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:34.432369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 465/931] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Oct 19 00:30:34 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:34.441489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:34.596512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775f8657-4074-4d06-b952-2c59d9168ea1 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:34.611597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:34.647949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfba2e1f-3116-43f0-a561-754c6193f450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:34.649391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfba2e1f-3116-43f0-a561-754c6193f450 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:34.650107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfba2e1f-3116-43f0-a561-754c6193f450 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:34.650683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfba2e1f-3116-43f0-a561-754c6193f450 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:34.765382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfba2e1f-3116-43f0-a561-754c6193f450 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:34.766961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfba2e1f-3116-43f0-a561-754c6193f450 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:34.929476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 467/932] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:30:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 00:30:34.936297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:34.982357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:35.079488 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e3e4ab-b09c-4e05-bbff-72afa7e49cb9 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:35.124997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:35.125644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:35.125925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:35.126234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:35.138006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:35.139540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:35.157851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:35.159033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:35.159947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:35.159947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:35.159947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:35.159947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 466/933] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:35.165907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfba2e1f-3116-43f0-a561-754c6193f450 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:35.184542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:35.300417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfba2e1f-3116-43f0-a561-754c6193f450 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:35.345843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:35.346682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:35.346984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:35.347613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:35.348433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:35.348868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:35.350626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:35.364919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 468/934] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:30:35.371674 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:35.479733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:35.499708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006de469-568a-4592-bb73-41c26e699a33 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:35.549152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:35.549803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:35.550079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:35.550349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] PATH_INFO: `/v3/projects/service` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:35.561124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:35.562997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:30:35.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:35.576466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:35.579632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 467/935] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:30:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:35.582827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:35.671914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:35.711542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dd6140-5c08-4d6e-a854-7f70931db54b None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:35.753581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013e5bf-1be9-4ded-b488-b90c01f20ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:35.754124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013e5bf-1be9-4ded-b488-b90c01f20ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:35.754437 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013e5bf-1be9-4ded-b488-b90c01f20ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:35.754872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013e5bf-1be9-4ded-b488-b90c01f20ad4 None admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:35.755724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013e5bf-1be9-4ded-b488-b90c01f20ad4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:35.756051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013e5bf-1be9-4ded-b488-b90c01f20ad4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:35.757712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013e5bf-1be9-4ded-b488-b90c01f20ad4 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:35.777957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 469/936] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:30:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:35.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:35.906839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f9613a-bd72-4268-a21c-09118d19a9ab None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:35.949849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2705941a-6dec-472b-b1da-f01694d84c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:35.950735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2705941a-6dec-472b-b1da-f01694d84c14 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:35.950735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2705941a-6dec-472b-b1da-f01694d84c14 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:35.951022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2705941a-6dec-472b-b1da-f01694d84c14 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:35.951748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2705941a-6dec-472b-b1da-f01694d84c14 None admin] RBAC: Adding query filter params (role.id=8b6c794a10cf4fcdb33edf34796d4819, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=93d2a83a28fd4608994317c2c7b6a723) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:35.952055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2705941a-6dec-472b-b1da-f01694d84c14 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:35.953485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2705941a-6dec-472b-b1da-f01694d84c14 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:35.971908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 468/937] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:30:35 2022] GET /identity/v3/role_assignments?user.id=93d2a83a28fd4608994317c2c7b6a723&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=8b6c794a10cf4fcdb33edf34796d4819 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:30:38.417649 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b7996-972f-44ec-8429-bd09976b64f5 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:38.418323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b7996-972f-44ec-8429-bd09976b64f5 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:38.418323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b7996-972f-44ec-8429-bd09976b64f5 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:38.420071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 470/938] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:30:38.432106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae758060-bd1e-41d7-a697-4eab3b87e85f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:38.432106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae758060-bd1e-41d7-a697-4eab3b87e85f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:38.432106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae758060-bd1e-41d7-a697-4eab3b87e85f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:38.455829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ae758060-bd1e-41d7-a697-4eab3b87e85f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:38.457210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ae758060-bd1e-41d7-a697-4eab3b87e85f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:38.463421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:38.591229 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 469/939] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:30:38.921432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2b7996-972f-44ec-8429-bd09976b64f5 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:39.047959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2b7996-972f-44ec-8429-bd09976b64f5 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:39.080869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:39.089168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e41c6c7-c5fe-4ab6-844d-6ae275a13246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:39.089903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e41c6c7-c5fe-4ab6-844d-6ae275a13246 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:39.090269 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e41c6c7-c5fe-4ab6-844d-6ae275a13246 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:39.090618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e41c6c7-c5fe-4ab6-844d-6ae275a13246 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:39.106715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e41c6c7-c5fe-4ab6-844d-6ae275a13246 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-2e41c6c7-c5fe-4ab6-844d-6ae275a13246 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:30:39.112589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:39.113776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:39.114915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 471/940] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:30:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:30:39.123058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae758060-bd1e-41d7-a697-4eab3b87e85f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:39.259873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae758060-bd1e-41d7-a697-4eab3b87e85f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:39.310288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:39.310983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:39.311261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:39.311532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:39.312231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:39.312529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:39.316624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:39.329400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 470/941] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:30:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:30:39.336058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e41c6c7-c5fe-4ab6-844d-6ae275a13246 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:39.476352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e41c6c7-c5fe-4ab6-844d-6ae275a13246 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:39.534610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:39.535206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:39.535481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:39.535758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:39.553187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:39.555582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:39.576406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:39.578565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:39.579822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:39.579822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:39.579822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:39.579822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 472/942] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:39.584367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:39.588179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:39.733079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6404ea-1854-44f8-85b8-59bcb1a4528b None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:39.787234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:39.788182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:39.788638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:39.789136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:39.790040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:39.790534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:39.793177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:39.814190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 471/943] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:39.822126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:39.883729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:39.955541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8f089-5404-409a-b21d-20478d0d5b35 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:40.004492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:40.006095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:40.006392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:40.006862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:40.025728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:40.027338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:30:40.040655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:40.041775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:40.042823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 473/944] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:39 2022] GET /identity/v3/users/placement => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:30:40.048269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:40.134535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:40.182972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15619168-1dfd-4408-841d-132efcdcf69c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:40.218955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:40.219574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:40.219849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:40.220118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:40.220845 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:40.221146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:40.222872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:40.245928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 472/945] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Oct 19 00:30:40 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:40.252048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:40.395479 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4b465-7678-47a5-a7be-d77ef81ad47f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:40.397580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:40.438055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:40.438773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:40.439053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:40.439323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:40.453165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:40.454694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:40.471918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:40.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:40.474631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:40.474631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:40.474631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:40.474631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 474/946] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:40.482936 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:40.631760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36274ee2-3175-4b44-8f5a-45c42ca7f647 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:40.659917 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:40.677722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:40.678432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:40.678849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:40.679199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:40.680569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:40.681050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:40.682496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:40.698622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 473/947] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:40.705528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:40.828184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7dd850-edba-4dc8-9ce0-0ab1e9129a2f None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:40.870517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:40.871246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:40.871560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:40.871813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:40.880087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:40.882168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:30:40.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:40.898640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:40.899694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 475/948] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:30:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:40.905925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:41.040654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a338b1-804f-46c7-afd2-b57a02746fbb None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:41.097973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50d145b-3142-46ab-8a59-2c6a7ac10864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:41.099721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50d145b-3142-46ab-8a59-2c6a7ac10864 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:41.100680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50d145b-3142-46ab-8a59-2c6a7ac10864 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:41.101154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50d145b-3142-46ab-8a59-2c6a7ac10864 None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:41.102624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50d145b-3142-46ab-8a59-2c6a7ac10864 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:41.103159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50d145b-3142-46ab-8a59-2c6a7ac10864 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:41.107239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50d145b-3142-46ab-8a59-2c6a7ac10864 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:41.132568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 474/949] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:30:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:41.139635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:41.156575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:41.258392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a4cc17-bbfc-4a18-892e-63ca0f218d77 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:41.303754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d393052d-df98-4a20-913c-73edab948c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:41.304385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393052d-df98-4a20-913c-73edab948c9f None admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:41.305214 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393052d-df98-4a20-913c-73edab948c9f None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:41.305508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393052d-df98-4a20-913c-73edab948c9f None admin] PATH_INFO: `/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/93d2a83a28fd4608994317c2c7b6a723/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:41.307594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393052d-df98-4a20-913c-73edab948c9f None admin] RBAC: Authorizing `identity:create_grant(project_id=fe3331f5cdee4ee6be88cac564cf0233, user_id=93d2a83a28fd4608994317c2c7b6a723, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:41.309241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393052d-df98-4a20-913c-73edab948c9f None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:41.323596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 476/950] 10.222.0.26 () {58 vars in 1439 bytes} [Wed Oct 19 00:30:41 2022] PUT /identity/v3/projects/fe3331f5cdee4ee6be88cac564cf0233/users/93d2a83a28fd4608994317c2c7b6a723/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:30:44.036571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3edb3-3938-4658-8f33-be944c90eff3 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:44.037438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3edb3-3938-4658-8f33-be944c90eff3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:44.037638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3edb3-3938-4658-8f33-be944c90eff3 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:44.040046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 475/951] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:30:44.048081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a1d103-ba2b-4c76-b318-2574c1ae3078 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:44.048897 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a1d103-ba2b-4c76-b318-2574c1ae3078 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:44.048897 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a1d103-ba2b-4c76-b318-2574c1ae3078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:44.074375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-f2a1d103-ba2b-4c76-b318-2574c1ae3078 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:44.076008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f2a1d103-ba2b-4c76-b318-2574c1ae3078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:44.163500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 477/952] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 00:30:44.505760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee3edb3-3938-4658-8f33-be944c90eff3 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:44.565134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:44.655235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee3edb3-3938-4658-8f33-be944c90eff3 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:44.707440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cd7efe-7e18-4eae-bb8a-b392665ff5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:44.708061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cd7efe-7e18-4eae-bb8a-b392665ff5cd None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:44.708342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cd7efe-7e18-4eae-bb8a-b392665ff5cd None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:44.708611 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cd7efe-7e18-4eae-bb8a-b392665ff5cd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:44.720925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cd7efe-7e18-4eae-bb8a-b392665ff5cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-10cd7efe-7e18-4eae-bb8a-b392665ff5cd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 00:30:44.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:44.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:44.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 476/953] 10.222.0.26 () {56 vars in 1096 bytes} [Wed Oct 19 00:30:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:30:44.734695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a1d103-ba2b-4c76-b318-2574c1ae3078 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:44.738922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:44.872041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a1d103-ba2b-4c76-b318-2574c1ae3078 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:44.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:44.932699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:44.932927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:44.933200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:44.933878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] RBAC: Adding query filter params (name=admin) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:44.934181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:44.935625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:44.945625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 478/954] 10.222.0.26 () {56 vars in 1099 bytes} [Wed Oct 19 00:30:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:44.953247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cd7efe-7e18-4eae-bb8a-b392665ff5cd None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:45.085003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:45.090463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cd7efe-7e18-4eae-bb8a-b392665ff5cd None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:45.130224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:45.130792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:45.131092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:45.131334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:45.143604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:45.145143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:45.167557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:45.168971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:45.168971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:45.168971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:45.168971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 477/955] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:45.176167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:45.265376 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:45.319136 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde87d3-1a55-475b-85b9-79a2b27321ce None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:45.358140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:45.358937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:45.359316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:45.359688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:45.360467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:45.360905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:45.362401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:45.376560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 479/956] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:45.382478 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:45.502833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c348754-19c8-470e-acbf-11cb58a81cee None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:45.556863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:45.557571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:45.557901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:45.558177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:45.568536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:45.571132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 00:30:45.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:45.590245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:45.592371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 478/957] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:45 2022] GET /identity/v3/users/placement => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 00:30:45.599729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:45.750981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d894a34-8c1c-41e3-bbda-df1991c31faf None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:45.793118 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:45.810066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:45.810731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:45.811101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:45.811459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:45.812373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:45.812699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:45.814652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:45.845611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 480/958] 10.222.0.26 () {56 vars in 1143 bytes} [Wed Oct 19 00:30:45 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:45.852100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:45.868924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:45.994972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf4d15a-3353-45f7-a71f-987019c1c33c None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:46.041503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ab7c4c-c095-41d8-ae61-a402e429d3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:46.042059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab7c4c-c095-41d8-ae61-a402e429d3ac None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:46.042405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab7c4c-c095-41d8-ae61-a402e429d3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:46.042719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab7c4c-c095-41d8-ae61-a402e429d3ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:46.141298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ab7c4c-c095-41d8-ae61-a402e429d3ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:46.144475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ab7c4c-c095-41d8-ae61-a402e429d3ac None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:46.165850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:46.289992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 479/959] 10.222.0.26 () {58 vars in 1282 bytes} [Wed Oct 19 00:30:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 00:30:46.298053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:46.409530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dba69e4-5b72-42d1-8d32-35914c2aed4c None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:46.448522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:46.449155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:46.449429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:46.449693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:46.461791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:46.463307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 00:30:46.479706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:46.481018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 00:30:46.482226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 00:30:46.482226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 00:30:46.482226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:46.485757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 481/960] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:30:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:46.486817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ab7c4c-c095-41d8-ae61-a402e429d3ac None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:46.598789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ab7c4c-c095-41d8-ae61-a402e429d3ac None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:46.613665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:46.638756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:46.639381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:46.639676 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:46.639965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:46.640625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:46.640924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:46.642300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:46.655550 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 480/961] 10.222.0.26 () {56 vars in 1109 bytes} [Wed Oct 19 00:30:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:46.661357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:46.704349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:46.774021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0b8b44-0126-49eb-b3b2-be84552b8915 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:46.809289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:46.809849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:46.810128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:46.810406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:46.817455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:46.818917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 00:30:46.831225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:46.832375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: ERROR keystone.server.flask.application Oct 19 00:30:46.833473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 482/962] 10.222.0.26 () {56 vars in 1111 bytes} [Wed Oct 19 00:30:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:46.838495 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:46.946204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafba186-ebd6-4fff-a622-e2511816acb5 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:46.981489 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45080ac2-5f5e-4fcb-8727-eaaf030863a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:46.982093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45080ac2-5f5e-4fcb-8727-eaaf030863a2 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:46.982368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45080ac2-5f5e-4fcb-8727-eaaf030863a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:46.982666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45080ac2-5f5e-4fcb-8727-eaaf030863a2 None admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:46.983360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45080ac2-5f5e-4fcb-8727-eaaf030863a2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:46.983660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45080ac2-5f5e-4fcb-8727-eaaf030863a2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:46.985076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45080ac2-5f5e-4fcb-8727-eaaf030863a2 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:46.998003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 481/963] 10.222.0.26 () {56 vars in 1148 bytes} [Wed Oct 19 00:30:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:47.004119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:47.098319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:47.106164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97843f23-fb16-4cf4-b2e3-b6c712ab7083 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:47.139581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f91e61-3ca6-4a4a-a2f0-7fda95bcd48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:47.140177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f91e61-3ca6-4a4a-a2f0-7fda95bcd48a None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:47.140459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f91e61-3ca6-4a4a-a2f0-7fda95bcd48a None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:47.140732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f91e61-3ca6-4a4a-a2f0-7fda95bcd48a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:47.141456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f91e61-3ca6-4a4a-a2f0-7fda95bcd48a None admin] RBAC: Adding query filter params (role.id=a68d428c5b8c41d69e3c1432c46d2dff, scope.project.id=fe3331f5cdee4ee6be88cac564cf0233, user.id=93d2a83a28fd4608994317c2c7b6a723) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:47.141761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f91e61-3ca6-4a4a-a2f0-7fda95bcd48a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:47.143226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f91e61-3ca6-4a4a-a2f0-7fda95bcd48a None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:47.149742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 483/964] 10.222.0.26 () {56 vars in 1374 bytes} [Wed Oct 19 00:30:47 2022] GET /identity/v3/role_assignments?user.id=93d2a83a28fd4608994317c2c7b6a723&scope.project.id=fe3331f5cdee4ee6be88cac564cf0233&role.id=a68d428c5b8c41d69e3c1432c46d2dff => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:49.327330 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c10d61-0a45-41c6-9f2d-6f9019cfc88b None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:49.327865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c10d61-0a45-41c6-9f2d-6f9019cfc88b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:49.327865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c10d61-0a45-41c6-9f2d-6f9019cfc88b None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:49.329252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 482/965] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:30:49.334458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e74c5-1e50-46de-8368-20870743159f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:49.334907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e74c5-1e50-46de-8368-20870743159f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:49.335046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e74c5-1e50-46de-8368-20870743159f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:49.353921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-654e74c5-1e50-46de-8368-20870743159f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:49.355155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-654e74c5-1e50-46de-8368-20870743159f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:49.421260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 484/966] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:30:49.685992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c10d61-0a45-41c6-9f2d-6f9019cfc88b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:49.717069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:49.788640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c10d61-0a45-41c6-9f2d-6f9019cfc88b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:49.822763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:49.823389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:49.823670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:49.823944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:49.824521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:49.825887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:49.837174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:49.838405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:49.839567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 483/967] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Oct 19 00:30:49 2022] GET /identity/v3/services/placement => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:49.844379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e74c5-1e50-46de-8368-20870743159f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:49.933308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:49.942561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e74c5-1e50-46de-8368-20870743159f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:49.977494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883dd0fb-aff1-4309-a61b-2661563fa6e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:49.978111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883dd0fb-aff1-4309-a61b-2661563fa6e3 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:49.978385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883dd0fb-aff1-4309-a61b-2661563fa6e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:49.978687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883dd0fb-aff1-4309-a61b-2661563fa6e3 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:49.979321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883dd0fb-aff1-4309-a61b-2661563fa6e3 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:49.979621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883dd0fb-aff1-4309-a61b-2661563fa6e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:49.980988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883dd0fb-aff1-4309-a61b-2661563fa6e3 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:49.986102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 485/968] 10.222.0.26 () {56 vars in 1116 bytes} [Wed Oct 19 00:30:49 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:49.992578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:50.112852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:50.119717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6615c4-8d4d-40e9-8ccf-5d692287ca36 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:50.154273 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa09f8c-86e5-4481-808e-4fbcc4cf0968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:50.154912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa09f8c-86e5-4481-808e-4fbcc4cf0968 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:50.155189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa09f8c-86e5-4481-808e-4fbcc4cf0968 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:50.155459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa09f8c-86e5-4481-808e-4fbcc4cf0968 None admin] PATH_INFO: `/v3/services` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:50.156079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa09f8c-86e5-4481-808e-4fbcc4cf0968 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:50.156378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa09f8c-86e5-4481-808e-4fbcc4cf0968 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:50.157734 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa09f8c-86e5-4481-808e-4fbcc4cf0968 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:50.162137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 484/969] 10.222.0.26 () {56 vars in 1116 bytes} [Wed Oct 19 00:30:49 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:52.391281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8e338-0b25-4509-9482-ef9cec2eb0a8 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:52.391822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8e338-0b25-4509-9482-ef9cec2eb0a8 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:52.391822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8e338-0b25-4509-9482-ef9cec2eb0a8 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:52.393169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 486/970] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:30:52.399708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d614f-461e-44e3-832b-3ad44c62345e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:52.400214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d614f-461e-44e3-832b-3ad44c62345e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:52.400304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d614f-461e-44e3-832b-3ad44c62345e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:52.419290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-591d614f-461e-44e3-832b-3ad44c62345e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:52.420536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-591d614f-461e-44e3-832b-3ad44c62345e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:52.489620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 485/971] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:30:52.761525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd8e338-0b25-4509-9482-ef9cec2eb0a8 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:52.870341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd8e338-0b25-4509-9482-ef9cec2eb0a8 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:52.905738 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7105f99-035c-4cc4-9883-5ebfd2c066e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:52.906339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7105f99-035c-4cc4-9883-5ebfd2c066e3 None admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:52.906644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7105f99-035c-4cc4-9883-5ebfd2c066e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:52.906917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7105f99-035c-4cc4-9883-5ebfd2c066e3 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:52.907560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7105f99-035c-4cc4-9883-5ebfd2c066e3 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:52.908917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7105f99-035c-4cc4-9883-5ebfd2c066e3 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:52.923843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 487/972] 10.222.0.26 () {60 vars in 1141 bytes} [Wed Oct 19 00:30:52 2022] POST /identity/v3/services => generated 246 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:30:55.095812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b54b99-bc8d-4b03-954f-a807a0abc6b2 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:55.096346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b54b99-bc8d-4b03-954f-a807a0abc6b2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:55.096346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b54b99-bc8d-4b03-954f-a807a0abc6b2 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:55.097957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 486/973] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:30:55.103915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ceb644-0c3f-4558-bb0e-b72539ac3e08 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:55.104359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ceb644-0c3f-4558-bb0e-b72539ac3e08 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:55.104449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ceb644-0c3f-4558-bb0e-b72539ac3e08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:55.122602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-34ceb644-0c3f-4558-bb0e-b72539ac3e08 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:55.123791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-34ceb644-0c3f-4558-bb0e-b72539ac3e08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:55.133033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:55.197402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 488/974] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:55 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:30:55.485247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b54b99-bc8d-4b03-954f-a807a0abc6b2 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:55.585245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:55.596548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b54b99-bc8d-4b03-954f-a807a0abc6b2 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:55.633100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:55.633751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:55.634036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:55.634308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:55.634932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:55.636328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:55.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:55.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:55.648035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 487/975] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Oct 19 00:30:55 2022] GET /identity/v3/services/placement => generated 90 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:55.653262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ceb644-0c3f-4558-bb0e-b72539ac3e08 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:55.752161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ceb644-0c3f-4558-bb0e-b72539ac3e08 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:55.786711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:55.787316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:55.787594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:55.787865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:55.788520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:55.788822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:55.790194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:55.795309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 489/976] 10.222.0.26 () {56 vars in 1116 bytes} [Wed Oct 19 00:30:55 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:55.801299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:55.900961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18315a2d-f74f-4efc-914c-7c5750ed5e97 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:55.934746 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7df68-2542-4fb7-b6f4-163102359e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:55.935349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7df68-2542-4fb7-b6f4-163102359e98 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:55.935628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7df68-2542-4fb7-b6f4-163102359e98 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:55.935898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7df68-2542-4fb7-b6f4-163102359e98 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:55.936590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7df68-2542-4fb7-b6f4-163102359e98 None admin] RBAC: Adding query filter params (interface=public, service_id=1ceea3ab0a144e66a7ef2059e1fd5895, region_id=RegionOne) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:55.936895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7df68-2542-4fb7-b6f4-163102359e98 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:55.938261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7df68-2542-4fb7-b6f4-163102359e98 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:55.943292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 488/977] 10.222.0.26 () {56 vars in 1251 bytes} [Wed Oct 19 00:30:55 2022] GET /identity/v3/endpoints?service_id=1ceea3ab0a144e66a7ef2059e1fd5895&interface=public®ion_id=RegionOne => generated 194 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:55.948982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:55.952736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:56.051030 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5af9f8-164c-4f68-9c8c-06d44625c111 None admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:56.084379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b247ebd-7352-4798-ac8f-c49d16b4a28b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:56.085046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b247ebd-7352-4798-ac8f-c49d16b4a28b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:56.085958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b247ebd-7352-4798-ac8f-c49d16b4a28b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:56.086244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b247ebd-7352-4798-ac8f-c49d16b4a28b None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:56.086875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b247ebd-7352-4798-ac8f-c49d16b4a28b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:56.088228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b247ebd-7352-4798-ac8f-c49d16b4a28b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:56.094976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 490/978] 10.222.0.26 () {56 vars in 1087 bytes} [Wed Oct 19 00:30:55 2022] GET /identity/v3/services => generated 2191 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:30:58.244762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8343c97-5051-42f0-95ee-21780821804c None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:58.245361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8343c97-5051-42f0-95ee-21780821804c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:58.245465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8343c97-5051-42f0-95ee-21780821804c None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:58.247095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 489/979] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:30:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:30:58.252398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d210a-b30c-4672-a4f6-0419e76288ea None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:58.252850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d210a-b30c-4672-a4f6-0419e76288ea None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:58.252956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d210a-b30c-4672-a4f6-0419e76288ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:58.271984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2e7d210a-b30c-4672-a4f6-0419e76288ea None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:30:58.273255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2e7d210a-b30c-4672-a4f6-0419e76288ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:30:58.337381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:58.343409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 491/980] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:30:58 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 00:30:58.610656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8343c97-5051-42f0-95ee-21780821804c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:58.633829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:58.725553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8343c97-5051-42f0-95ee-21780821804c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:58.765253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:58.765889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:58.766171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:58.766437 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:58.767057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:58.768430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 00:30:58.778721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 00:30:58.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application with Lock( Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self._enter() Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application return self.creator() Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application created_value = creator( Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: ERROR keystone.server.flask.application Oct 19 00:30:58.781313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 490/981] 10.222.0.26 () {56 vars in 1117 bytes} [Wed Oct 19 00:30:58 2022] GET /identity/v3/services/placement => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 00:30:58.786017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7d210a-b30c-4672-a4f6-0419e76288ea None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:58.889332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7d210a-b30c-4672-a4f6-0419e76288ea None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:58.923581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ac3058-d2bb-4392-ace5-39df5b37943b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:58.924184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ac3058-d2bb-4392-ace5-39df5b37943b None admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:58.924458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ac3058-d2bb-4392-ace5-39df5b37943b None admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:58.924727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ac3058-d2bb-4392-ace5-39df5b37943b None admin] PATH_INFO: `/v3/services` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:58.925376 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ac3058-d2bb-4392-ace5-39df5b37943b None admin] RBAC: Adding query filter params (name=placement) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:30:58.925672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ac3058-d2bb-4392-ace5-39df5b37943b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:58.927092 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ac3058-d2bb-4392-ace5-39df5b37943b None admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:58.932461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 492/982] 10.222.0.26 () {56 vars in 1116 bytes} [Wed Oct 19 00:30:58 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:30:58.938270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:30:59.020821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:30:59.039726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a96d0-dfe4-4dcb-95fc-0ef7fcc8d8d4 None admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:30:59.072628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3da65f2-756c-4503-889e-7a25f1770428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:30:59.073226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3da65f2-756c-4503-889e-7a25f1770428 None admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:30:59.073502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3da65f2-756c-4503-889e-7a25f1770428 None admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:30:59.073771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3da65f2-756c-4503-889e-7a25f1770428 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:30:59.075097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3da65f2-756c-4503-889e-7a25f1770428 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:30:59.075860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3da65f2-756c-4503-889e-7a25f1770428 None admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:30:59.094758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 491/983] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:30:58 2022] POST /identity/v3/endpoints => generated 331 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:31:08.930110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=69 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:31:09.093452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:07.821087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752337e9-b408-4df6-aac5-d534e82ea441 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:07.821624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752337e9-b408-4df6-aac5-d534e82ea441 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:07.821624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752337e9-b408-4df6-aac5-d534e82ea441 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:07.827069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 493/984] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:07.828421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc8b04e-7477-4498-a9ee-4b785f1b3946 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:07.828860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc8b04e-7477-4498-a9ee-4b785f1b3946 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:07.829076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc8b04e-7477-4498-a9ee-4b785f1b3946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:08.003966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-cdc8b04e-7477-4498-a9ee-4b785f1b3946 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:08.006927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cdc8b04e-7477-4498-a9ee-4b785f1b3946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:08.267319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 492/985] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:08.565679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb3d15-ee77-4a06-a42c-e2dd7dc919ef None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:08.566207 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb3d15-ee77-4a06-a42c-e2dd7dc919ef None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:08.566827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb3d15-ee77-4a06-a42c-e2dd7dc919ef None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:08.570729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 494/986] 10.222.0.26 () {54 vars in 964 bytes} [Wed Oct 19 00:32:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:08.577264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006bbcdb-0188-4d15-b913-a4a519899e2e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:08.577485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006bbcdb-0188-4d15-b913-a4a519899e2e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:08.577880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006bbcdb-0188-4d15-b913-a4a519899e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:08.679987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-006bbcdb-0188-4d15-b913-a4a519899e2e None None] MFA Rules not processed for user `5756f4ff604a413b8af1f12c82cc4870`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:08.681335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-006bbcdb-0188-4d15-b913-a4a519899e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:08.709269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:08.784289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 493/987] 10.222.0.26 () {58 vars in 1061 bytes} [Wed Oct 19 00:32:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:32:08.794036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffb3d15-ee77-4a06-a42c-e2dd7dc919ef None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:08.796018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bffb3d15-ee77-4a06-a42c-e2dd7dc919ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:08.938036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffb3d15-ee77-4a06-a42c-e2dd7dc919ef None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:09.043725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6531a5-4664-479c-81c7-9afc04cca498 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:09.044533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6531a5-4664-479c-81c7-9afc04cca498 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:09.044965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6531a5-4664-479c-81c7-9afc04cca498 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:09.045249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6531a5-4664-479c-81c7-9afc04cca498 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:09.047208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0c6531a5-4664-479c-81c7-9afc04cca498 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:09.128903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:09.230124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6531a5-4664-479c-81c7-9afc04cca498 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:09.233006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6531a5-4664-479c-81c7-9afc04cca498 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:09.273345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 495/988] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:11.852849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c4994f-33d8-4c3b-883f-f46ad81ee464 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:11.853429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c4994f-33d8-4c3b-883f-f46ad81ee464 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:11.853429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c4994f-33d8-4c3b-883f-f46ad81ee464 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:11.854772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 494/989] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:11.860571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4dd22a-e92c-4162-9f5c-342e32d3c830 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:11.861089 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4dd22a-e92c-4162-9f5c-342e32d3c830 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:11.861089 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4dd22a-e92c-4162-9f5c-342e32d3c830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:11.939891 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-fb4dd22a-e92c-4162-9f5c-342e32d3c830 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:11.941107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-fb4dd22a-e92c-4162-9f5c-342e32d3c830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:12.047805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 496/990] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:12.356650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c4994f-33d8-4c3b-883f-f46ad81ee464 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:12.358191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c3c4994f-33d8-4c3b-883f-f46ad81ee464 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:12.478677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c4994f-33d8-4c3b-883f-f46ad81ee464 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:12.573489 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:12.576739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1cb7e4-e80e-40a5-b32d-1e199347f76b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:12.577791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1cb7e4-e80e-40a5-b32d-1e199347f76b service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:12.578332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1cb7e4-e80e-40a5-b32d-1e199347f76b service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:12.578713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1cb7e4-e80e-40a5-b32d-1e199347f76b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:12.582280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fe1cb7e4-e80e-40a5-b32d-1e199347f76b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:12.708101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1cb7e4-e80e-40a5-b32d-1e199347f76b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:12.709763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1cb7e4-e80e-40a5-b32d-1e199347f76b service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:12.734951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 495/991] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:15.323900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d302d45b-1a5e-415e-b2d6-ff01e990c954 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:15.324435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d302d45b-1a5e-415e-b2d6-ff01e990c954 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:15.324435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d302d45b-1a5e-415e-b2d6-ff01e990c954 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:15.325793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 497/992] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:15.331817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ccfab-1f1b-4215-a675-a660701b305d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:15.332302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ccfab-1f1b-4215-a675-a660701b305d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:15.332509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3ccfab-1f1b-4215-a675-a660701b305d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:15.413901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1b3ccfab-1f1b-4215-a675-a660701b305d None None] MFA Rules not processed for user `a43f8e46711349d4bcf7bab7f618f326`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:15.415274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1b3ccfab-1f1b-4215-a675-a660701b305d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:15.471350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:15.528585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 496/993] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:15 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:15.865311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d302d45b-1a5e-415e-b2d6-ff01e990c954 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:15.867129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d302d45b-1a5e-415e-b2d6-ff01e990c954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:15.878095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:16.008696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d302d45b-1a5e-415e-b2d6-ff01e990c954 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:16.119085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca35ad85-37b0-4716-9eee-ea3b7399b524 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:16.119605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca35ad85-37b0-4716-9eee-ea3b7399b524 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:16.119885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca35ad85-37b0-4716-9eee-ea3b7399b524 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:16.120153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca35ad85-37b0-4716-9eee-ea3b7399b524 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:16.121862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ca35ad85-37b0-4716-9eee-ea3b7399b524 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:16.189185 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:16.246817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca35ad85-37b0-4716-9eee-ea3b7399b524 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:16.248739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca35ad85-37b0-4716-9eee-ea3b7399b524 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:16.271635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 498/994] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:15 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:32:19.228350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd09f6-2aa5-4231-8cec-932105d13b6d None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:19.228903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd09f6-2aa5-4231-8cec-932105d13b6d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:19.228903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd09f6-2aa5-4231-8cec-932105d13b6d None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:19.230208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 497/995] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:19.235889 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b16686a-e774-4908-9e1c-4303ca16177b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:19.236338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b16686a-e774-4908-9e1c-4303ca16177b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:19.236427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b16686a-e774-4908-9e1c-4303ca16177b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:19.319635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7b16686a-e774-4908-9e1c-4303ca16177b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:19.321029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7b16686a-e774-4908-9e1c-4303ca16177b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:19.437155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 499/996] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:19.774832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1df91-1ba1-43e4-9c59-31b22a8db56a None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:19.775256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1df91-1ba1-43e4-9c59-31b22a8db56a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:19.775567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c1df91-1ba1-43e4-9c59-31b22a8db56a None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:19.776907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 498/997] 10.222.0.26 () {54 vars in 964 bytes} [Wed Oct 19 00:32:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:32:19.783919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5adbe49-6bcc-4f1f-8080-a53b7e877f7c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:19.784223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5adbe49-6bcc-4f1f-8080-a53b7e877f7c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:19.784527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5adbe49-6bcc-4f1f-8080-a53b7e877f7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:19.811501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:19.878587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c5adbe49-6bcc-4f1f-8080-a53b7e877f7c None None] MFA Rules not processed for user `5756f4ff604a413b8af1f12c82cc4870`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:19.880327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c5adbe49-6bcc-4f1f-8080-a53b7e877f7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:19.971625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 500/998] 10.222.0.26 () {58 vars in 1061 bytes} [Wed Oct 19 00:32:19 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:19.983690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c1df91-1ba1-43e4-9c59-31b22a8db56a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:19.985379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-02c1df91-1ba1-43e4-9c59-31b22a8db56a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:20.120610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c1df91-1ba1-43e4-9c59-31b22a8db56a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:20.212723 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c91ff-8be8-48e2-884b-9ab339aebfe8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:20.213344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c91ff-8be8-48e2-884b-9ab339aebfe8 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:20.213617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c91ff-8be8-48e2-884b-9ab339aebfe8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:20.213885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c91ff-8be8-48e2-884b-9ab339aebfe8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:20.215859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-317c91ff-8be8-48e2-884b-9ab339aebfe8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:20.225244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:20.353679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c91ff-8be8-48e2-884b-9ab339aebfe8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:20.355292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c91ff-8be8-48e2-884b-9ab339aebfe8 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:20.376616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 499/999] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:23.961389 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a9a754-8e75-4cb8-8257-3e620765f510 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:23.961948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a9a754-8e75-4cb8-8257-3e620765f510 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:23.961948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a9a754-8e75-4cb8-8257-3e620765f510 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:23.963288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 501/1000] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:23.968790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdfab5-3d8a-47f2-a3ad-7945b942da08 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:23.969290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdfab5-3d8a-47f2-a3ad-7945b942da08 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:23.969512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdfab5-3d8a-47f2-a3ad-7945b942da08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:24.045690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-07bdfab5-3d8a-47f2-a3ad-7945b942da08 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:24.047018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-07bdfab5-3d8a-47f2-a3ad-7945b942da08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:24.123936 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:24.166594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 500/1001] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:24.452330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a9a754-8e75-4cb8-8257-3e620765f510 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:24.453881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-08a9a754-8e75-4cb8-8257-3e620765f510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:24.562687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a9a754-8e75-4cb8-8257-3e620765f510 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:24.577085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:24.640059 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68b3221-afb9-417b-8077-02a2a8b3cddf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:24.640550 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b3221-afb9-417b-8077-02a2a8b3cddf service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:24.640825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b3221-afb9-417b-8077-02a2a8b3cddf service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:24.641094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b3221-afb9-417b-8077-02a2a8b3cddf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:24.642720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a68b3221-afb9-417b-8077-02a2a8b3cddf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:24.753301 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68b3221-afb9-417b-8077-02a2a8b3cddf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:24.755050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68b3221-afb9-417b-8077-02a2a8b3cddf service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:24.773858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 502/1002] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:27.026143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4225c85d-9d1e-4c75-b2d0-88cecefbcc34 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:27.026800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4225c85d-9d1e-4c75-b2d0-88cecefbcc34 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:27.026800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4225c85d-9d1e-4c75-b2d0-88cecefbcc34 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:27.028182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 501/1003] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:27.033490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc98afc0-6c64-4fb8-b0ed-b2f0180e1ee4 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:27.034067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc98afc0-6c64-4fb8-b0ed-b2f0180e1ee4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:27.034067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc98afc0-6c64-4fb8-b0ed-b2f0180e1ee4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:27.095091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:27.122099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-dc98afc0-6c64-4fb8-b0ed-b2f0180e1ee4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:27.123608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-dc98afc0-6c64-4fb8-b0ed-b2f0180e1ee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:27.233409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 503/1004] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:27.521607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4225c85d-9d1e-4c75-b2d0-88cecefbcc34 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:27.523284 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4225c85d-9d1e-4c75-b2d0-88cecefbcc34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:27.639438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4225c85d-9d1e-4c75-b2d0-88cecefbcc34 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:27.717385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa9e67e-6760-4742-9a98-a760784d3251 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:27.717994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa9e67e-6760-4742-9a98-a760784d3251 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:27.718267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa9e67e-6760-4742-9a98-a760784d3251 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:27.718563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa9e67e-6760-4742-9a98-a760784d3251 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:27.720638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-dfa9e67e-6760-4742-9a98-a760784d3251 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:27.738853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:27.836184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa9e67e-6760-4742-9a98-a760784d3251 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:27.837536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa9e67e-6760-4742-9a98-a760784d3251 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:27.864340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 502/1005] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:30.220701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f994ba-9850-46b7-ae1d-7d94599879f3 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:30.221260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f994ba-9850-46b7-ae1d-7d94599879f3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:30.221260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f994ba-9850-46b7-ae1d-7d94599879f3 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:30.223217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 504/1006] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:30.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a09c6-c85a-45a8-8ecf-51438ff77f32 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:30.228945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a09c6-c85a-45a8-8ecf-51438ff77f32 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:30.229084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a09c6-c85a-45a8-8ecf-51438ff77f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:30.297983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-146a09c6-c85a-45a8-8ecf-51438ff77f32 None None] MFA Rules not processed for user `a43f8e46711349d4bcf7bab7f618f326`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:30.299175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-146a09c6-c85a-45a8-8ecf-51438ff77f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:30.366655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:30.402576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 503/1007] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:30 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:30.685563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f994ba-9850-46b7-ae1d-7d94599879f3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:30.687197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-84f994ba-9850-46b7-ae1d-7d94599879f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:30.800192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f994ba-9850-46b7-ae1d-7d94599879f3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:30.822149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:30.880910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d7086e-2558-4a67-8a1a-df33a6631af1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:30.881523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d7086e-2558-4a67-8a1a-df33a6631af1 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:30.881810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d7086e-2558-4a67-8a1a-df33a6631af1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:30.882082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d7086e-2558-4a67-8a1a-df33a6631af1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:30.883758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c2d7086e-2558-4a67-8a1a-df33a6631af1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:30.996447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d7086e-2558-4a67-8a1a-df33a6631af1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:30.997896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d7086e-2558-4a67-8a1a-df33a6631af1 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:31.016912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 505/1008] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:30 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:33.531365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f506d2a4-7ffb-4518-8867-2c815e3a6dcf None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:33.532044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f506d2a4-7ffb-4518-8867-2c815e3a6dcf None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:33.532044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f506d2a4-7ffb-4518-8867-2c815e3a6dcf None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:33.533411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 504/1009] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:33.539748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ebb20-fdf4-48b4-a0b7-412b077416fd None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:33.540234 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ebb20-fdf4-48b4-a0b7-412b077416fd None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:33.540358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ebb20-fdf4-48b4-a0b7-412b077416fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:33.616754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:33.619835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-3b2ebb20-fdf4-48b4-a0b7-412b077416fd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:33.621610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3b2ebb20-fdf4-48b4-a0b7-412b077416fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:33.738925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 506/1010] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:34.035835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f506d2a4-7ffb-4518-8867-2c815e3a6dcf None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:34.037752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f506d2a4-7ffb-4518-8867-2c815e3a6dcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:34.150070 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f506d2a4-7ffb-4518-8867-2c815e3a6dcf None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:34.226256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb9acf0-c667-4f35-bb4a-fadf9e4cc5b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:34.226889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9acf0-c667-4f35-bb4a-fadf9e4cc5b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:34.227164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9acf0-c667-4f35-bb4a-fadf9e4cc5b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:34.227492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9acf0-c667-4f35-bb4a-fadf9e4cc5b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:34.229041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fdb9acf0-c667-4f35-bb4a-fadf9e4cc5b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:34.258698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:34.342278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb9acf0-c667-4f35-bb4a-fadf9e4cc5b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:34.343750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb9acf0-c667-4f35-bb4a-fadf9e4cc5b3 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:34.363021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 505/1011] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:36.617076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869cd049-f4af-4f18-8fda-c36dcf5d248c None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:36.617649 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869cd049-f4af-4f18-8fda-c36dcf5d248c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:36.617755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869cd049-f4af-4f18-8fda-c36dcf5d248c None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:36.619217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 507/1012] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:32:36.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada9319c-7d60-41ff-a42d-820d03da890d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:36.625439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada9319c-7d60-41ff-a42d-820d03da890d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:36.625597 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada9319c-7d60-41ff-a42d-820d03da890d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:36.695413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ada9319c-7d60-41ff-a42d-820d03da890d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:36.696645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ada9319c-7d60-41ff-a42d-820d03da890d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:36.781333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:36.804275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 506/1013] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:37.096837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869cd049-f4af-4f18-8fda-c36dcf5d248c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:37.100007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-869cd049-f4af-4f18-8fda-c36dcf5d248c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:37.226499 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869cd049-f4af-4f18-8fda-c36dcf5d248c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:37.259289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:37.310867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee8b972-3875-435b-864f-00c5530b6e00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:37.311482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8b972-3875-435b-864f-00c5530b6e00 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:37.311751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8b972-3875-435b-864f-00c5530b6e00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:37.312039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8b972-3875-435b-864f-00c5530b6e00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:37.313636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-dee8b972-3875-435b-864f-00c5530b6e00 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:37.427813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee8b972-3875-435b-864f-00c5530b6e00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:37.429153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee8b972-3875-435b-864f-00c5530b6e00 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:37.447586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 508/1014] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:40.139458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be217cad-51a3-487d-8346-60f773ac9d7e None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:40.140025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be217cad-51a3-487d-8346-60f773ac9d7e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:40.140025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be217cad-51a3-487d-8346-60f773ac9d7e None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:40.141375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 507/1015] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:40.147401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c349959c-ecd6-4932-a3da-0ae29b168af9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:40.147930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c349959c-ecd6-4932-a3da-0ae29b168af9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:40.148583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c349959c-ecd6-4932-a3da-0ae29b168af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:40.226597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c349959c-ecd6-4932-a3da-0ae29b168af9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:40.227841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c349959c-ecd6-4932-a3da-0ae29b168af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:40.236651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:40.334216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 509/1016] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:40 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:40.618833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be217cad-51a3-487d-8346-60f773ac9d7e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:40.620395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-be217cad-51a3-487d-8346-60f773ac9d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:40.731869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be217cad-51a3-487d-8346-60f773ac9d7e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:40.811773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a104272-4087-4f25-9008-34ae387b6f33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:40.812388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a104272-4087-4f25-9008-34ae387b6f33 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:40.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a104272-4087-4f25-9008-34ae387b6f33 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:40.812941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a104272-4087-4f25-9008-34ae387b6f33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:40.814576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6a104272-4087-4f25-9008-34ae387b6f33 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:40.852563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:40.925644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a104272-4087-4f25-9008-34ae387b6f33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:40.927014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a104272-4087-4f25-9008-34ae387b6f33 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:40.945022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 508/1017] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:40 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:45.118696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d457e0ce-5db5-4ad1-9344-11517478dddc None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:45.119219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d457e0ce-5db5-4ad1-9344-11517478dddc None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:45.119219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d457e0ce-5db5-4ad1-9344-11517478dddc None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:45.120651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 510/1018] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:32:45.125627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cee97f-652f-4cce-91ff-0f7476200b34 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:45.126063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cee97f-652f-4cce-91ff-0f7476200b34 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:45.126178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cee97f-652f-4cce-91ff-0f7476200b34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:45.195905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-81cee97f-652f-4cce-91ff-0f7476200b34 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:45.197022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-81cee97f-652f-4cce-91ff-0f7476200b34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:45.290677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:45.302713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 509/1019] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:45.595046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d457e0ce-5db5-4ad1-9344-11517478dddc None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:45.596589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d457e0ce-5db5-4ad1-9344-11517478dddc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:45.703369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d457e0ce-5db5-4ad1-9344-11517478dddc None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:45.744390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:45.782605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3e730f-bac2-4a5d-9190-89f17843d9a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:45.783212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e730f-bac2-4a5d-9190-89f17843d9a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:45.783490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e730f-bac2-4a5d-9190-89f17843d9a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:45.783757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3e730f-bac2-4a5d-9190-89f17843d9a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:45.785992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2b3e730f-bac2-4a5d-9190-89f17843d9a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:45.908076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3e730f-bac2-4a5d-9190-89f17843d9a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:45.909433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3e730f-bac2-4a5d-9190-89f17843d9a6 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:45.929685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 511/1020] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:32:48.798224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e9ac8a-8914-4be1-adbb-92481b002d5b None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:48.798817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e9ac8a-8914-4be1-adbb-92481b002d5b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:48.798817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e9ac8a-8914-4be1-adbb-92481b002d5b None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:48.800411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 510/1021] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:48.805693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30059ebc-3dca-42a8-919b-6516479cad4f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:48.806167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30059ebc-3dca-42a8-919b-6516479cad4f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:48.806261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30059ebc-3dca-42a8-919b-6516479cad4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:48.875405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-30059ebc-3dca-42a8-919b-6516479cad4f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:48.876604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-30059ebc-3dca-42a8-919b-6516479cad4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:48.896280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:48.976991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 512/1022] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:49.261680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e9ac8a-8914-4be1-adbb-92481b002d5b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:49.264003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-26e9ac8a-8914-4be1-adbb-92481b002d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:49.379628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e9ac8a-8914-4be1-adbb-92481b002d5b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:49.458120 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f6ed1-3e27-4abf-b32f-0ee3d4611824 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:49.458816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f6ed1-3e27-4abf-b32f-0ee3d4611824 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:49.459098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f6ed1-3e27-4abf-b32f-0ee3d4611824 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:49.459373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f6ed1-3e27-4abf-b32f-0ee3d4611824 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:49.461007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-249f6ed1-3e27-4abf-b32f-0ee3d4611824 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:49.507525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:49.585570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f6ed1-3e27-4abf-b32f-0ee3d4611824 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:49.587091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f6ed1-3e27-4abf-b32f-0ee3d4611824 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:49.607923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 511/1023] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:49 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:32:54.230863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b986e3-e3dc-4646-aac7-c11309624482 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:54.231392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b986e3-e3dc-4646-aac7-c11309624482 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:54.231392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b986e3-e3dc-4646-aac7-c11309624482 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:54.232768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 513/1024] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:32:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:32:54.238683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0080c-3455-43b4-bb3b-3142e1872ff6 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:54.239241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0080c-3455-43b4-bb3b-3142e1872ff6 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:54.239241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0080c-3455-43b4-bb3b-3142e1872ff6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:54.321504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-96c0080c-3455-43b4-bb3b-3142e1872ff6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:32:54.323658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-96c0080c-3455-43b4-bb3b-3142e1872ff6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:54.435678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:54.438540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 512/1025] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:32:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:32:54.726177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b986e3-e3dc-4646-aac7-c11309624482 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:32:54.727812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-41b986e3-e3dc-4646-aac7-c11309624482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:54.843855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b986e3-e3dc-4646-aac7-c11309624482 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:32:54.894854 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:32:54.929410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10017b28-6a2c-47a9-b0d4-105bf5d88df3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:32:54.930401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10017b28-6a2c-47a9-b0d4-105bf5d88df3 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:32:54.930933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10017b28-6a2c-47a9-b0d4-105bf5d88df3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:32:54.931387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10017b28-6a2c-47a9-b0d4-105bf5d88df3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:32:54.934517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-10017b28-6a2c-47a9-b0d4-105bf5d88df3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:32:55.048657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10017b28-6a2c-47a9-b0d4-105bf5d88df3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:32:55.050632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10017b28-6a2c-47a9-b0d4-105bf5d88df3 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:32:55.069730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 514/1026] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:32:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:00.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca417b2b-a3a3-445d-9912-197d58bfe682 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:00.215682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca417b2b-a3a3-445d-9912-197d58bfe682 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:00.216241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca417b2b-a3a3-445d-9912-197d58bfe682 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:00.219863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 513/1027] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:33:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:00.228606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5eaf84-4af7-4447-b923-b44e900a60eb None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:00.229138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5eaf84-4af7-4447-b923-b44e900a60eb None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:00.229344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5eaf84-4af7-4447-b923-b44e900a60eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:00.375931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7e5eaf84-4af7-4447-b923-b44e900a60eb None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:00.377381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7e5eaf84-4af7-4447-b923-b44e900a60eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:00.407318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:00.516642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 515/1028] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:33:00 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:04.804161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280118e-f665-456f-9681-8589799718f6 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:04.804161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280118e-f665-456f-9681-8589799718f6 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:04.804962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280118e-f665-456f-9681-8589799718f6 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:04.805859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 514/1029] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:33:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:04.811274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0a7fa-aab3-43ec-a61d-10e50e00b257 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:04.811820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0a7fa-aab3-43ec-a61d-10e50e00b257 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:04.812053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0a7fa-aab3-43ec-a61d-10e50e00b257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:04.896744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-53d0a7fa-aab3-43ec-a61d-10e50e00b257 None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:04.898145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-53d0a7fa-aab3-43ec-a61d-10e50e00b257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:04.978604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:05.022459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 516/1030] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:33:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:06.546210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f165a5-3fe6-4c03-906c-3a18da2ece46 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:06.546779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f165a5-3fe6-4c03-906c-3a18da2ece46 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:06.546779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f165a5-3fe6-4c03-906c-3a18da2ece46 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:06.548364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 515/1031] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:33:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:06.554397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797d832-3761-443b-b66b-2c98dd8e3a93 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:06.554893 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797d832-3761-443b-b66b-2c98dd8e3a93 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:06.554997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797d832-3761-443b-b66b-2c98dd8e3a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:06.639593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8797d832-3761-443b-b66b-2c98dd8e3a93 None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:06.640821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8797d832-3761-443b-b66b-2c98dd8e3a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:06.776573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 517/1032] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:33:06 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:10.568876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e1253-db92-47d0-a289-2ad73d4b8b19 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:10.569388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e1253-db92-47d0-a289-2ad73d4b8b19 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:10.569388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e1253-db92-47d0-a289-2ad73d4b8b19 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:10.571032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 516/1033] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:10.580847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35061032-eec6-4073-85bb-8f54be0fe5c6 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:10.581447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35061032-eec6-4073-85bb-8f54be0fe5c6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:10.581584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35061032-eec6-4073-85bb-8f54be0fe5c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:10.643474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:10.713676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-35061032-eec6-4073-85bb-8f54be0fe5c6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:10.715813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-35061032-eec6-4073-85bb-8f54be0fe5c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:10.857514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 518/1034] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:11.218514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa6c43-ce05-425f-8ac0-3e1a35bd17b5 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:11.219089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa6c43-ce05-425f-8ac0-3e1a35bd17b5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:11.219656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa6c43-ce05-425f-8ac0-3e1a35bd17b5 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:11.221244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 517/1035] 10.222.0.26 () {54 vars in 961 bytes} [Wed Oct 19 00:33:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:11.229832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7025af4c-f411-4411-8546-b5ec9d48c4a1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:11.230422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7025af4c-f411-4411-8546-b5ec9d48c4a1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:11.230939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7025af4c-f411-4411-8546-b5ec9d48c4a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:11.320869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7025af4c-f411-4411-8546-b5ec9d48c4a1 None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:11.322223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7025af4c-f411-4411-8546-b5ec9d48c4a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:11.373588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:11.448953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 519/1036] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Oct 19 00:33:11 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:11.458906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fa6c43-ce05-425f-8ac0-3e1a35bd17b5 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:11.461680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-58fa6c43-ce05-425f-8ac0-3e1a35bd17b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:11.676504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fa6c43-ce05-425f-8ac0-3e1a35bd17b5 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:11.779278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:11.854141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e071b1-2ca3-4117-8460-aa020634b9fe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:11.855365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e071b1-2ca3-4117-8460-aa020634b9fe service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:11.855955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e071b1-2ca3-4117-8460-aa020634b9fe service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:11.856549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e071b1-2ca3-4117-8460-aa020634b9fe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:11.859738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-87e071b1-2ca3-4117-8460-aa020634b9fe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:12.054167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e071b1-2ca3-4117-8460-aa020634b9fe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:12.057852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e071b1-2ca3-4117-8460-aa020634b9fe service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:12.097166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 518/1037] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 639 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:15.053863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6a96c0-7042-4209-9f03-901b38efa1f4 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:15.054334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6a96c0-7042-4209-9f03-901b38efa1f4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:15.054624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6a96c0-7042-4209-9f03-901b38efa1f4 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:15.055777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 520/1038] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:15.063004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc1cd4d-becd-43cb-9272-b5989298f654 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:15.063508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc1cd4d-becd-43cb-9272-b5989298f654 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:15.064578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc1cd4d-becd-43cb-9272-b5989298f654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:15.190077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ddc1cd4d-becd-43cb-9272-b5989298f654 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:15.192111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ddc1cd4d-becd-43cb-9272-b5989298f654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:15.252429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:15.335652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 519/1039] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:15.789896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6a96c0-7042-4209-9f03-901b38efa1f4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:15.791454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-fb6a96c0-7042-4209-9f03-901b38efa1f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:16.134210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6a96c0-7042-4209-9f03-901b38efa1f4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:16.152842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:16.378884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52704064-0f73-4783-acc7-730c70adc885 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:16.379922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52704064-0f73-4783-acc7-730c70adc885 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:16.380404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52704064-0f73-4783-acc7-730c70adc885 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:16.380896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52704064-0f73-4783-acc7-730c70adc885 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:16.384019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-52704064-0f73-4783-acc7-730c70adc885 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:16.534718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52704064-0f73-4783-acc7-730c70adc885 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:16.537069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52704064-0f73-4783-acc7-730c70adc885 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:16.579966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 521/1040] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 790 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:33:19.261190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c6a4d-16b3-4d07-a3bb-f49d46bbca34 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:19.261764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c6a4d-16b3-4d07-a3bb-f49d46bbca34 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:19.262366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c6a4d-16b3-4d07-a3bb-f49d46bbca34 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:19.266267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 520/1041] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:19 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:19.272183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978fc58-9db3-451a-9a34-ad05e3834714 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:19.273232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978fc58-9db3-451a-9a34-ad05e3834714 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:19.273905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978fc58-9db3-451a-9a34-ad05e3834714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:19.292862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:19.358385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-1978fc58-9db3-451a-9a34-ad05e3834714 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:19.359664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1978fc58-9db3-451a-9a34-ad05e3834714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:19.478534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 522/1042] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:19.827707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f76c34-3c1c-4297-b220-8a7ad6810e75 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:19.827959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f76c34-3c1c-4297-b220-8a7ad6810e75 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:19.828217 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f76c34-3c1c-4297-b220-8a7ad6810e75 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:19.829687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 521/1043] 10.222.0.26 () {54 vars in 961 bytes} [Wed Oct 19 00:33:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:33:19.840329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbbc52e-b212-4453-9ff6-6ef2f9653439 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:19.840834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbbc52e-b212-4453-9ff6-6ef2f9653439 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:19.841327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbbc52e-b212-4453-9ff6-6ef2f9653439 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:19.992122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-bdbbc52e-b212-4453-9ff6-6ef2f9653439 None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:19.994028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bdbbc52e-b212-4453-9ff6-6ef2f9653439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:20.037612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:20.184021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 523/1044] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Oct 19 00:33:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 346 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:20.194657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f76c34-3c1c-4297-b220-8a7ad6810e75 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:20.196635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-48f76c34-3c1c-4297-b220-8a7ad6810e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:20.514853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f76c34-3c1c-4297-b220-8a7ad6810e75 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:20.564965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:20.663148 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ed91f-0d4f-4b6e-a074-3262e11b92ed None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:20.663751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ed91f-0d4f-4b6e-a074-3262e11b92ed None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:20.663751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66ed91f-0d4f-4b6e-a074-3262e11b92ed None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:20.665491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 524/1045] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:20.671012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ed804f-3ad3-49d7-98d5-c9ec6a43c529 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:20.671366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ed804f-3ad3-49d7-98d5-c9ec6a43c529 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:20.671567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ed804f-3ad3-49d7-98d5-c9ec6a43c529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:20.712406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8ca1dd-f836-418a-8723-54230056b77a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:20.713019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ca1dd-f836-418a-8723-54230056b77a service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:20.713345 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ca1dd-f836-418a-8723-54230056b77a service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:20.713620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ca1dd-f836-418a-8723-54230056b77a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:20.715323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9b8ca1dd-f836-418a-8723-54230056b77a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:20.777882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d5ed804f-3ad3-49d7-98d5-c9ec6a43c529 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:20.779025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d5ed804f-3ad3-49d7-98d5-c9ec6a43c529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:20.860108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8ca1dd-f836-418a-8723-54230056b77a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:20.862869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8ca1dd-f836-418a-8723-54230056b77a service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:20.890669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 522/1046] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 696 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:20.927457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:20.946573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 525/1047] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:33:21.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4650c44-3490-4871-aa97-25cb635f1b9a None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:21.248986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4650c44-3490-4871-aa97-25cb635f1b9a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:21.249353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4650c44-3490-4871-aa97-25cb635f1b9a None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:21.251032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 523/1048] 10.222.0.26 () {54 vars in 963 bytes} [Wed Oct 19 00:33:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:21.256618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b120d57-e6a5-4894-b630-ba6ad22b0d94 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:21.256975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b120d57-e6a5-4894-b630-ba6ad22b0d94 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:21.257297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b120d57-e6a5-4894-b630-ba6ad22b0d94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:21.351361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8b120d57-e6a5-4894-b630-ba6ad22b0d94 None None] MFA Rules not processed for user `13270581a86e430886c878ff84302764`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:21.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8b120d57-e6a5-4894-b630-ba6ad22b0d94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:21.450791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 526/1049] 10.222.0.26 () {58 vars in 1060 bytes} [Wed Oct 19 00:33:21 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:21.454525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4650c44-3490-4871-aa97-25cb635f1b9a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:21.456891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a4650c44-3490-4871-aa97-25cb635f1b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:21.490446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:21.632344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4650c44-3490-4871-aa97-25cb635f1b9a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:21.751462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10acf777-20ac-46e9-a638-dbbd93b34c1d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:21.752783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10acf777-20ac-46e9-a638-dbbd93b34c1d service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:21.753074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10acf777-20ac-46e9-a638-dbbd93b34c1d service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:21.753384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10acf777-20ac-46e9-a638-dbbd93b34c1d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:21.755351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-10acf777-20ac-46e9-a638-dbbd93b34c1d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:21.887234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:21.929770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10acf777-20ac-46e9-a638-dbbd93b34c1d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:21.931800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10acf777-20ac-46e9-a638-dbbd93b34c1d service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:21.952064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 524/1050] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:23.221748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c79433-805c-4a68-8f27-6ce38e2d0c58 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:23.222934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c79433-805c-4a68-8f27-6ce38e2d0c58 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:23.223215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c79433-805c-4a68-8f27-6ce38e2d0c58 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:23.224629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 527/1051] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:33:23.230203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21b89e-c8fa-4356-9b5f-851395581f6f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:23.230740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21b89e-c8fa-4356-9b5f-851395581f6f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:23.230897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f21b89e-c8fa-4356-9b5f-851395581f6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:23.306429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1f21b89e-c8fa-4356-9b5f-851395581f6f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:23.307703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1f21b89e-c8fa-4356-9b5f-851395581f6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:23.417002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 525/1052] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:23.716388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c79433-805c-4a68-8f27-6ce38e2d0c58 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:23.717914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a9c79433-805c-4a68-8f27-6ce38e2d0c58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:23.823702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:23.906730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c79433-805c-4a68-8f27-6ce38e2d0c58 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:24.055927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edebb32c-527e-41b5-9090-6529d3abd024 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:24.058440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edebb32c-527e-41b5-9090-6529d3abd024 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:24.058440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edebb32c-527e-41b5-9090-6529d3abd024 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:24.058440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edebb32c-527e-41b5-9090-6529d3abd024 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:24.061189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-edebb32c-527e-41b5-9090-6529d3abd024 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:24.139931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:24.195598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edebb32c-527e-41b5-9090-6529d3abd024 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:24.197279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edebb32c-527e-41b5-9090-6529d3abd024 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:24.218664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 528/1053] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:33:26.550788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca3d12f-9632-4275-9304-08a25d04f061 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:26.551342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca3d12f-9632-4275-9304-08a25d04f061 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:26.551342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca3d12f-9632-4275-9304-08a25d04f061 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:26.554441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 526/1054] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:26.559926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8995b1f-a181-4de0-83bc-8e62032613e9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:26.560348 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8995b1f-a181-4de0-83bc-8e62032613e9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:26.560532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8995b1f-a181-4de0-83bc-8e62032613e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:26.638558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e8995b1f-a181-4de0-83bc-8e62032613e9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:26.641600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e8995b1f-a181-4de0-83bc-8e62032613e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:26.754535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 529/1055] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:27.071440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca3d12f-9632-4275-9304-08a25d04f061 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:27.074654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4ca3d12f-9632-4275-9304-08a25d04f061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:27.135569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:27.290476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca3d12f-9632-4275-9304-08a25d04f061 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:27.429149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:27.435854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0282e4e1-4859-4835-9552-5b4dc5fe4447 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:27.437034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0282e4e1-4859-4835-9552-5b4dc5fe4447 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:27.437634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0282e4e1-4859-4835-9552-5b4dc5fe4447 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:27.438374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0282e4e1-4859-4835-9552-5b4dc5fe4447 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:27.440984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0282e4e1-4859-4835-9552-5b4dc5fe4447 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:27.585391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0282e4e1-4859-4835-9552-5b4dc5fe4447 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:27.588760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0282e4e1-4859-4835-9552-5b4dc5fe4447 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:27.613730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 527/1056] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 543 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:29.977393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d0ec4-823c-4460-aa9c-c3fb37867af1 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:29.978124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d0ec4-823c-4460-aa9c-c3fb37867af1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:29.978124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d0ec4-823c-4460-aa9c-c3fb37867af1 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:29.979310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 530/1057] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:29.985185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4eaf15-0280-472c-b9dd-8d6ee584ba9a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:29.985692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4eaf15-0280-472c-b9dd-8d6ee584ba9a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:29.985801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4eaf15-0280-472c-b9dd-8d6ee584ba9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:30.071544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-3a4eaf15-0280-472c-b9dd-8d6ee584ba9a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:30.072750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3a4eaf15-0280-472c-b9dd-8d6ee584ba9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:30.169645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:30.213575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 528/1058] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:30.520009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d0ec4-823c-4460-aa9c-c3fb37867af1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:30.522981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-de1d0ec4-823c-4460-aa9c-c3fb37867af1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:30.572044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:30.731494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d0ec4-823c-4460-aa9c-c3fb37867af1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:30.864744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:30.889982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52709ef-1111-4cfa-9931-8735afdb62e7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:30.890853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52709ef-1111-4cfa-9931-8735afdb62e7 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:30.891298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52709ef-1111-4cfa-9931-8735afdb62e7 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:30.891683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52709ef-1111-4cfa-9931-8735afdb62e7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:30.894293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f52709ef-1111-4cfa-9931-8735afdb62e7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:31.018310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52709ef-1111-4cfa-9931-8735afdb62e7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:31.020278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52709ef-1111-4cfa-9931-8735afdb62e7 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:31.041719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 531/1059] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:33.532221 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89054053-1255-421a-9f7f-a780ad1b5fdf None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:33.532768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89054053-1255-421a-9f7f-a780ad1b5fdf None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:33.533653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89054053-1255-421a-9f7f-a780ad1b5fdf None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:33.535669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 529/1060] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:33.541530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a906068d-6bbc-4045-bc5c-6f7acee868e7 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:33.542010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a906068d-6bbc-4045-bc5c-6f7acee868e7 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:33.542093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a906068d-6bbc-4045-bc5c-6f7acee868e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:33.616084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a906068d-6bbc-4045-bc5c-6f7acee868e7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:33.617319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a906068d-6bbc-4045-bc5c-6f7acee868e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:33.683788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:33.740526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 532/1061] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:33:34.045764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89054053-1255-421a-9f7f-a780ad1b5fdf None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:34.047542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-89054053-1255-421a-9f7f-a780ad1b5fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:34.216244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89054053-1255-421a-9f7f-a780ad1b5fdf None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:34.278503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:34.359778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9180e28-c9d6-4d0f-8700-2fad63da41d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:34.360494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9180e28-c9d6-4d0f-8700-2fad63da41d3 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:34.360863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9180e28-c9d6-4d0f-8700-2fad63da41d3 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:34.361315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9180e28-c9d6-4d0f-8700-2fad63da41d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:34.364528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b9180e28-c9d6-4d0f-8700-2fad63da41d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:34.507369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9180e28-c9d6-4d0f-8700-2fad63da41d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:34.509262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9180e28-c9d6-4d0f-8700-2fad63da41d3 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:34.534327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 530/1062] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:36.821085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f8e88-f660-41fe-bcb4-e57efb7285e3 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:36.821668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f8e88-f660-41fe-bcb4-e57efb7285e3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:36.821668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f8e88-f660-41fe-bcb4-e57efb7285e3 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:36.822917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 533/1063] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:36.828353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7146d03a-34e7-49d8-b9ab-ef42ca43c525 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:36.828947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7146d03a-34e7-49d8-b9ab-ef42ca43c525 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:36.828947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7146d03a-34e7-49d8-b9ab-ef42ca43c525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:36.900944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-7146d03a-34e7-49d8-b9ab-ef42ca43c525 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:36.902379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7146d03a-34e7-49d8-b9ab-ef42ca43c525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:36.911150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:37.005348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 531/1064] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:37.299834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f8e88-f660-41fe-bcb4-e57efb7285e3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:37.301385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-745f8e88-f660-41fe-bcb4-e57efb7285e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:37.460952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f8e88-f660-41fe-bcb4-e57efb7285e3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:37.510111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:37.585010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f80d51a-ae02-407c-ac19-c5c6da191f50 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:37.585899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80d51a-ae02-407c-ac19-c5c6da191f50 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:37.586178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80d51a-ae02-407c-ac19-c5c6da191f50 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:37.586431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80d51a-ae02-407c-ac19-c5c6da191f50 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:37.588061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1f80d51a-ae02-407c-ac19-c5c6da191f50 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:37.685037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f80d51a-ae02-407c-ac19-c5c6da191f50 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:37.686689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f80d51a-ae02-407c-ac19-c5c6da191f50 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:37.707125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 534/1065] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:38.081877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc485ef-19c6-470e-b65d-ce91763e1578 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:38.082521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc485ef-19c6-470e-b65d-ce91763e1578 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:38.082521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc485ef-19c6-470e-b65d-ce91763e1578 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:38.083767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 532/1066] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:38.089633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6258cd6-c293-45e5-9437-24ee45fa5eed None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:38.090153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6258cd6-c293-45e5-9437-24ee45fa5eed None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:38.090153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6258cd6-c293-45e5-9437-24ee45fa5eed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:38.165339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:38.169201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e6258cd6-c293-45e5-9437-24ee45fa5eed None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:38.171194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e6258cd6-c293-45e5-9437-24ee45fa5eed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:38.288567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 535/1067] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:38.597801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d4763d-a29c-4116-843e-58ab1df6ce53 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:38.598102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d4763d-a29c-4116-843e-58ab1df6ce53 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:38.598377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d4763d-a29c-4116-843e-58ab1df6ce53 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:38.599764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 533/1068] 10.222.0.26 () {54 vars in 962 bytes} [Wed Oct 19 00:33:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:38.609087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb561d-8087-4e76-a644-a20afea0f7f0 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:38.609372 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb561d-8087-4e76-a644-a20afea0f7f0 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:38.609652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfb561d-8087-4e76-a644-a20afea0f7f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:38.709078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-0bfb561d-8087-4e76-a644-a20afea0f7f0 None None] MFA Rules not processed for user `7e3f6be584e34bfb9e430c7756d071d5`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:38.710362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0bfb561d-8087-4e76-a644-a20afea0f7f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:38.804001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:38.812299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 536/1069] 10.222.0.26 () {58 vars in 1059 bytes} [Wed Oct 19 00:33:38 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:38.821599 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d4763d-a29c-4116-843e-58ab1df6ce53 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:38.823277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-91d4763d-a29c-4116-843e-58ab1df6ce53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:38.953355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d4763d-a29c-4116-843e-58ab1df6ce53 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:38.998971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:39.042585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5552093-7544-4c18-93dd-c8d2ec543d2b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e3f6be584e34bfb9e430c7756d071d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:39.043220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5552093-7544-4c18-93dd-c8d2ec543d2b service glance] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:39.043495 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5552093-7544-4c18-93dd-c8d2ec543d2b service glance] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:39.043762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5552093-7544-4c18-93dd-c8d2ec543d2b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:39.045489 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a5552093-7544-4c18-93dd-c8d2ec543d2b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:39.179688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5552093-7544-4c18-93dd-c8d2ec543d2b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:39.181062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5552093-7544-4c18-93dd-c8d2ec543d2b service glance] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:39.206028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 534/1070] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:40.000902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70edc8bc-da18-48f2-b0f1-4bc637182d9e None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:40.001382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70edc8bc-da18-48f2-b0f1-4bc637182d9e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:40.001586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70edc8bc-da18-48f2-b0f1-4bc637182d9e None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:40.002964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 537/1071] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:40.009448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfbafdf-7f48-45a1-bf4b-993c44f233ec None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:40.009735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfbafdf-7f48-45a1-bf4b-993c44f233ec None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:40.010013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfbafdf-7f48-45a1-bf4b-993c44f233ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:40.015951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c9053-4d11-4326-8972-ab3c093de8ac None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:40.016331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c9053-4d11-4326-8972-ab3c093de8ac None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:40.016671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c9053-4d11-4326-8972-ab3c093de8ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:40.086570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8cfbafdf-7f48-45a1-bf4b-993c44f233ec None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:40.087802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8cfbafdf-7f48-45a1-bf4b-993c44f233ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:40.094987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-391c9053-4d11-4326-8972-ab3c093de8ac None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:40.096997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-391c9053-4d11-4326-8972-ab3c093de8ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:40.107936 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:40.193276 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 538/1072] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:33:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:40.201978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 535/1073] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:40.238751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9074dc0-29f3-484e-8fa7-5944e5e958a4 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:40.239302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9074dc0-29f3-484e-8fa7-5944e5e958a4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:40.239918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9074dc0-29f3-484e-8fa7-5944e5e958a4 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:40.241847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 539/1074] 10.222.0.26 () {54 vars in 943 bytes} [Wed Oct 19 00:33:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:40.248940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39de048-1fcf-4650-b289-486eeb841a25 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:40.249224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39de048-1fcf-4650-b289-486eeb841a25 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:40.249555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39de048-1fcf-4650-b289-486eeb841a25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:40.339474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c39de048-1fcf-4650-b289-486eeb841a25 None None] MFA Rules not processed for user `33b9c777f8284edeb758b00de044fd45`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:40.340690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c39de048-1fcf-4650-b289-486eeb841a25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:40.431745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:40.462868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 536/1075] 10.222.0.26 () {58 vars in 1040 bytes} [Wed Oct 19 00:33:40 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:33:40.473102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9074dc0-29f3-484e-8fa7-5944e5e958a4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:40.474652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a9074dc0-29f3-484e-8fa7-5944e5e958a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:40.497686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39de048-1fcf-4650-b289-486eeb841a25 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:40.498948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c39de048-1fcf-4650-b289-486eeb841a25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:40.617152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:40.633573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39de048-1fcf-4650-b289-486eeb841a25 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:40.675682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9074dc0-29f3-484e-8fa7-5944e5e958a4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:40.747774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:40.760249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81589fa-932e-4710-a3a3-f604220a113e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:40.761178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81589fa-932e-4710-a3a3-f604220a113e service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:40.761459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81589fa-932e-4710-a3a3-f604220a113e service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:40.761735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81589fa-932e-4710-a3a3-f604220a113e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:40.763850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e81589fa-932e-4710-a3a3-f604220a113e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:40.803417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8600b9-f952-4d4b-9dfe-c1b3a734b6d2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:40.804032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8600b9-f952-4d4b-9dfe-c1b3a734b6d2 service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:40.804311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8600b9-f952-4d4b-9dfe-c1b3a734b6d2 service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:40.804592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8600b9-f952-4d4b-9dfe-c1b3a734b6d2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:40.806377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4d8600b9-f952-4d4b-9dfe-c1b3a734b6d2 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:40.865495 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81589fa-932e-4710-a3a3-f604220a113e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:40.867302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81589fa-932e-4710-a3a3-f604220a113e service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:40.882377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:40.887705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 537/1076] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:33:40.899554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8600b9-f952-4d4b-9dfe-c1b3a734b6d2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:40.901392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8600b9-f952-4d4b-9dfe-c1b3a734b6d2 service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:40.911312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 540/1077] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:33:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:33:43.268523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d163be4-2227-4664-8711-d0e18fabcf1e None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:43.269099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d163be4-2227-4664-8711-d0e18fabcf1e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:43.269099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d163be4-2227-4664-8711-d0e18fabcf1e None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:43.270590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 538/1078] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:33:43.276138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd9d170-b845-4a86-893a-3eaf6fa03980 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:43.276635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd9d170-b845-4a86-893a-3eaf6fa03980 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:43.276744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd9d170-b845-4a86-893a-3eaf6fa03980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:43.353849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ddd9d170-b845-4a86-893a-3eaf6fa03980 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:43.355104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ddd9d170-b845-4a86-893a-3eaf6fa03980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:43.478529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 541/1079] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:43.775618 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d163be4-2227-4664-8711-d0e18fabcf1e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:43.779213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5d163be4-2227-4664-8711-d0e18fabcf1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:43.915257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:43.961064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d163be4-2227-4664-8711-d0e18fabcf1e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:44.095651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd85ee9-8266-44be-9688-dd5e69271ca8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:44.099837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd85ee9-8266-44be-9688-dd5e69271ca8 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:44.100170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd85ee9-8266-44be-9688-dd5e69271ca8 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:44.100382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd85ee9-8266-44be-9688-dd5e69271ca8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:44.102177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3cd85ee9-8266-44be-9688-dd5e69271ca8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:44.202179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:44.216281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd85ee9-8266-44be-9688-dd5e69271ca8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:44.217895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd85ee9-8266-44be-9688-dd5e69271ca8 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:44.242356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 539/1080] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:46.557384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ac57b-641d-4457-9cbe-c195ab49d1c3 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:46.557978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ac57b-641d-4457-9cbe-c195ab49d1c3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:46.557978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ac57b-641d-4457-9cbe-c195ab49d1c3 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:46.559473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 542/1081] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:46.564762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2e00c2-f26a-46a3-b943-040b4943b1cc None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:46.565241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2e00c2-f26a-46a3-b943-040b4943b1cc None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:46.565344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2e00c2-f26a-46a3-b943-040b4943b1cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:46.642496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5e2e00c2-f26a-46a3-b943-040b4943b1cc None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:46.643889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5e2e00c2-f26a-46a3-b943-040b4943b1cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:46.765198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 540/1082] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:47.058578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1ac57b-641d-4457-9cbe-c195ab49d1c3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:47.060117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4b1ac57b-641d-4457-9cbe-c195ab49d1c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:47.136218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:47.234652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1ac57b-641d-4457-9cbe-c195ab49d1c3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:47.366468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e74163a-aa8f-432c-b935-cac43966292b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:47.367152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74163a-aa8f-432c-b935-cac43966292b service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:47.367430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74163a-aa8f-432c-b935-cac43966292b service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:47.367704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74163a-aa8f-432c-b935-cac43966292b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:47.369325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1e74163a-aa8f-432c-b935-cac43966292b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:47.400728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:47.472948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e74163a-aa8f-432c-b935-cac43966292b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:47.474739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e74163a-aa8f-432c-b935-cac43966292b service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:47.498344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 543/1083] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:49.747467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b90552c-91ca-4903-aa0f-4f5fa1743fdc None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:49.748045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b90552c-91ca-4903-aa0f-4f5fa1743fdc None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:49.748045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b90552c-91ca-4903-aa0f-4f5fa1743fdc None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:49.749350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 541/1084] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:33:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:33:49.754988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a08dd00-5ba5-4a78-b076-263e19e29bf9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:49.755516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a08dd00-5ba5-4a78-b076-263e19e29bf9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:49.755516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a08dd00-5ba5-4a78-b076-263e19e29bf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:49.836679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-9a08dd00-5ba5-4a78-b076-263e19e29bf9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:33:49.837850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9a08dd00-5ba5-4a78-b076-263e19e29bf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:49.949469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 544/1085] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:33:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:33:50.253463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b90552c-91ca-4903-aa0f-4f5fa1743fdc None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:33:50.255208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3b90552c-91ca-4903-aa0f-4f5fa1743fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:50.334176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:50.453443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b90552c-91ca-4903-aa0f-4f5fa1743fdc None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:33:50.592736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca7f47b-1569-4525-92b2-c635b5cf97a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:33:50.593507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca7f47b-1569-4525-92b2-c635b5cf97a1 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:33:50.593786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca7f47b-1569-4525-92b2-c635b5cf97a1 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:33:50.594054 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca7f47b-1569-4525-92b2-c635b5cf97a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:33:50.596102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3ca7f47b-1569-4525-92b2-c635b5cf97a1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:33:50.619867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:33:50.701724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca7f47b-1569-4525-92b2-c635b5cf97a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:33:50.703516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca7f47b-1569-4525-92b2-c635b5cf97a1 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:33:50.725763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 542/1086] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:33:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:33:59.948200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:00.723202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=38 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:04.035564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0eb553-2dcc-4d93-8d13-3b09a7930bec None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:04.036323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0eb553-2dcc-4d93-8d13-3b09a7930bec None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:04.036323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0eb553-2dcc-4d93-8d13-3b09a7930bec None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:04.037445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 545/1087] 10.222.0.26 () {54 vars in 918 bytes} [Wed Oct 19 00:34:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:04.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0075cf78-e5a1-4f42-a34e-1e8c85675a5f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:04.043420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0075cf78-e5a1-4f42-a34e-1e8c85675a5f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:04.043562 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0075cf78-e5a1-4f42-a34e-1e8c85675a5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:04.124996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0075cf78-e5a1-4f42-a34e-1e8c85675a5f None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:04.126880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0075cf78-e5a1-4f42-a34e-1e8c85675a5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:04.255509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 543/1088] 10.222.0.26 () {58 vars in 1015 bytes} [Wed Oct 19 00:34:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:04.271010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63536ae1-39ad-4525-b5f4-9b2d356a0fb8 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:04.271289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63536ae1-39ad-4525-b5f4-9b2d356a0fb8 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:04.271562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63536ae1-39ad-4525-b5f4-9b2d356a0fb8 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:04.272964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 546/1089] 10.222.0.26 () {54 vars in 918 bytes} [Wed Oct 19 00:34:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:04.278566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efae699-846e-4e0e-8e72-d0b82555a09b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:04.278855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efae699-846e-4e0e-8e72-d0b82555a09b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:04.279137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efae699-846e-4e0e-8e72-d0b82555a09b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:04.369277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:04.372102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1efae699-846e-4e0e-8e72-d0b82555a09b None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:04.373879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1efae699-846e-4e0e-8e72-d0b82555a09b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:04.506389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 544/1090] 10.222.0.26 () {58 vars in 1015 bytes} [Wed Oct 19 00:34:04 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:07.160562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9915b98-e2c5-4acc-9ffc-4dc57f59dd95 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:07.161140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9915b98-e2c5-4acc-9ffc-4dc57f59dd95 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:07.161140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9915b98-e2c5-4acc-9ffc-4dc57f59dd95 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:07.162419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 547/1091] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:07.168003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4819f7-bf0b-4378-9efc-7047c070c1b7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:07.168454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4819f7-bf0b-4378-9efc-7047c070c1b7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:07.168564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4819f7-bf0b-4378-9efc-7047c070c1b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:07.239253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-7f4819f7-bf0b-4378-9efc-7047c070c1b7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:07.240482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7f4819f7-bf0b-4378-9efc-7047c070c1b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:07.299664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:07.343885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 545/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:07.640401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9915b98-e2c5-4acc-9ffc-4dc57f59dd95 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:07.642562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a9915b98-e2c5-4acc-9ffc-4dc57f59dd95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:07.797958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9915b98-e2c5-4acc-9ffc-4dc57f59dd95 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:07.874790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:07.918817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528aba55-140a-4ace-add8-5536517e3e6b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:07.919424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528aba55-140a-4ace-add8-5536517e3e6b service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:07.919732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528aba55-140a-4ace-add8-5536517e3e6b service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:07.920009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528aba55-140a-4ace-add8-5536517e3e6b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:07.921616 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-528aba55-140a-4ace-add8-5536517e3e6b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:08.021078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528aba55-140a-4ace-add8-5536517e3e6b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:08.022715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528aba55-140a-4ace-add8-5536517e3e6b service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:08.042109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 548/1093] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:09.224577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a0a672-ea79-4fd9-8e1e-85d810b68885 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:09.225116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a0a672-ea79-4fd9-8e1e-85d810b68885 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:09.225116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a0a672-ea79-4fd9-8e1e-85d810b68885 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:09.226449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 546/1094] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:34:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:09.232818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1116fceb-7263-41cf-b7b2-04d90d18f560 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:09.233293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1116fceb-7263-41cf-b7b2-04d90d18f560 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:09.233408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1116fceb-7263-41cf-b7b2-04d90d18f560 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:09.318666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-1116fceb-7263-41cf-b7b2-04d90d18f560 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:09.319900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1116fceb-7263-41cf-b7b2-04d90d18f560 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:09.348564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:09.426086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 549/1095] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:34:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:09.580357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a0a672-ea79-4fd9-8e1e-85d810b68885 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:09.582982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f0a0a672-ea79-4fd9-8e1e-85d810b68885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:09.765792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a0a672-ea79-4fd9-8e1e-85d810b68885 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:09.810834 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:09.894687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88682c3-480e-4ed0-9c76-16b86984a5d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:09.895304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88682c3-480e-4ed0-9c76-16b86984a5d1 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:09.895577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88682c3-480e-4ed0-9c76-16b86984a5d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:09.895843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88682c3-480e-4ed0-9c76-16b86984a5d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:09.897457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a88682c3-480e-4ed0-9c76-16b86984a5d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:10.009659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88682c3-480e-4ed0-9c76-16b86984a5d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:10.011366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88682c3-480e-4ed0-9c76-16b86984a5d1 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:10.037708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 547/1096] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:11.214865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88df6e6e-d9b4-49ec-8abc-7bcd42f9154b None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:11.215526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88df6e6e-d9b4-49ec-8abc-7bcd42f9154b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:11.215526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88df6e6e-d9b4-49ec-8abc-7bcd42f9154b None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:11.216765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 550/1097] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:34:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:11.225583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ab894-bd5a-4836-984c-6f372acd6974 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:11.226165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ab894-bd5a-4836-984c-6f372acd6974 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:11.226517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ab894-bd5a-4836-984c-6f372acd6974 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:11.290613 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:11.319148 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5e6ab894-bd5a-4836-984c-6f372acd6974 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:11.320550 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5e6ab894-bd5a-4836-984c-6f372acd6974 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:11.431867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 548/1098] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:34:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:11.589398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88df6e6e-d9b4-49ec-8abc-7bcd42f9154b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:11.590981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-88df6e6e-d9b4-49ec-8abc-7bcd42f9154b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:11.785356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88df6e6e-d9b4-49ec-8abc-7bcd42f9154b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:11.799589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:11.964240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78b24bc-6c25-4777-a338-d55625f5ef68 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:11.964850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b24bc-6c25-4777-a338-d55625f5ef68 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:11.965125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b24bc-6c25-4777-a338-d55625f5ef68 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:11.965391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b24bc-6c25-4777-a338-d55625f5ef68 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:11.967086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c78b24bc-6c25-4777-a338-d55625f5ef68 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:12.125095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78b24bc-6c25-4777-a338-d55625f5ef68 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:12.129266 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78b24bc-6c25-4777-a338-d55625f5ef68 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:12.158364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 551/1099] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 569 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:13.367137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2d7109-6e9e-4374-bba4-4243104918ca None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:13.367759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2d7109-6e9e-4374-bba4-4243104918ca None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:13.367920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2d7109-6e9e-4374-bba4-4243104918ca None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:13.369474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 549/1100] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:34:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:34:13.376703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c07dd-dc6a-41ad-b7b7-fe17787a2f09 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:13.377305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c07dd-dc6a-41ad-b7b7-fe17787a2f09 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:13.377510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c07dd-dc6a-41ad-b7b7-fe17787a2f09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:13.399024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:13.477524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-cf5c07dd-dc6a-41ad-b7b7-fe17787a2f09 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:13.478867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-cf5c07dd-dc6a-41ad-b7b7-fe17787a2f09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:13.612126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 552/1101] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:34:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:34:13.767503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2d7109-6e9e-4374-bba4-4243104918ca None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:13.769263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ce2d7109-6e9e-4374-bba4-4243104918ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:13.891775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:13.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2d7109-6e9e-4374-bba4-4243104918ca None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:14.066034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b726b731-325c-403e-8fb6-3cf770504fdd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:14.066794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b726b731-325c-403e-8fb6-3cf770504fdd service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:14.067120 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b726b731-325c-403e-8fb6-3cf770504fdd service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:14.067435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b726b731-325c-403e-8fb6-3cf770504fdd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:14.069164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b726b731-325c-403e-8fb6-3cf770504fdd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:14.161080 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:14.176663 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b726b731-325c-403e-8fb6-3cf770504fdd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:14.178262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b726b731-325c-403e-8fb6-3cf770504fdd service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:14.198989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 550/1102] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:34:15.504953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b424726-42b2-41ab-9960-a77b0e4c6a4d None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:15.505570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b424726-42b2-41ab-9960-a77b0e4c6a4d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:15.505570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b424726-42b2-41ab-9960-a77b0e4c6a4d None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:15.506785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 553/1103] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:34:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:15.514865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd8763d-13be-44a2-a760-ef30b46fd1d5 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:15.515428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd8763d-13be-44a2-a760-ef30b46fd1d5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:15.515736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd8763d-13be-44a2-a760-ef30b46fd1d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:15.616627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1dd8763d-13be-44a2-a760-ef30b46fd1d5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:15.617865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1dd8763d-13be-44a2-a760-ef30b46fd1d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:15.733654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 551/1104] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:34:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:15.904773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b424726-42b2-41ab-9960-a77b0e4c6a4d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:15.907029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7b424726-42b2-41ab-9960-a77b0e4c6a4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:16.026424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:16.106848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b424726-42b2-41ab-9960-a77b0e4c6a4d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:16.265493 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f658b-98ca-4ba2-856b-f154885865df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:16.267482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f658b-98ca-4ba2-856b-f154885865df service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:16.267822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f658b-98ca-4ba2-856b-f154885865df service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:16.268140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f658b-98ca-4ba2-856b-f154885865df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:16.270208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-975f658b-98ca-4ba2-856b-f154885865df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:16.332446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:16.383108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f658b-98ca-4ba2-856b-f154885865df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:16.384722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f658b-98ca-4ba2-856b-f154885865df service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:16.404651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 554/1105] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:34:17.540010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad6087-a762-405e-8511-ae2358074395 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:17.540598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad6087-a762-405e-8511-ae2358074395 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:17.540598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad6087-a762-405e-8511-ae2358074395 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:17.541964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 552/1106] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:34:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:34:17.553335 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92b2d3a-cc4c-4042-9404-5f7238c17480 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:17.554155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92b2d3a-cc4c-4042-9404-5f7238c17480 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:17.554339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92b2d3a-cc4c-4042-9404-5f7238c17480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:17.639364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a92b2d3a-cc4c-4042-9404-5f7238c17480 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:17.640592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a92b2d3a-cc4c-4042-9404-5f7238c17480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:17.747731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 555/1107] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:34:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:17.904841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad6087-a762-405e-8511-ae2358074395 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:17.906377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-daad6087-a762-405e-8511-ae2358074395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:17.963990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:18.077359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad6087-a762-405e-8511-ae2358074395 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:18.207752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebe2319-e90b-4856-8af8-24737e3829a5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:18.208383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebe2319-e90b-4856-8af8-24737e3829a5 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:18.208654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebe2319-e90b-4856-8af8-24737e3829a5 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:18.208926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebe2319-e90b-4856-8af8-24737e3829a5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:18.210625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-eebe2319-e90b-4856-8af8-24737e3829a5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:18.220210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:18.317377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eebe2319-e90b-4856-8af8-24737e3829a5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:18.319032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eebe2319-e90b-4856-8af8-24737e3829a5 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:18.338847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 553/1108] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:19.434094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a4d78-2d6b-41d3-983d-74e8a7ef4cab None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:19.434704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a4d78-2d6b-41d3-983d-74e8a7ef4cab None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:19.434704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a4d78-2d6b-41d3-983d-74e8a7ef4cab None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:19.435991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 556/1109] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:34:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:19.443918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12eab-55ba-440c-93f4-cf826d9275e0 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:19.444454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12eab-55ba-440c-93f4-cf826d9275e0 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:19.444572 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12eab-55ba-440c-93f4-cf826d9275e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:19.532276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-40f12eab-55ba-440c-93f4-cf826d9275e0 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:19.533725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-40f12eab-55ba-440c-93f4-cf826d9275e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:19.606748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:19.638173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 554/1110] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:34:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:19.793463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a4d78-2d6b-41d3-983d-74e8a7ef4cab None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:19.794572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2a9a4d78-2d6b-41d3-983d-74e8a7ef4cab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:19.812257 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:19.959730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a4d78-2d6b-41d3-983d-74e8a7ef4cab None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:20.090266 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:20.115063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a005f1-a2ea-4aa1-97ea-d32acaded046 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:20.115682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a005f1-a2ea-4aa1-97ea-d32acaded046 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:20.115962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a005f1-a2ea-4aa1-97ea-d32acaded046 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:20.116233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a005f1-a2ea-4aa1-97ea-d32acaded046 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:20.117848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-90a005f1-a2ea-4aa1-97ea-d32acaded046 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:20.211091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a005f1-a2ea-4aa1-97ea-d32acaded046 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:20.212708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a005f1-a2ea-4aa1-97ea-d32acaded046 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:20.231769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 557/1111] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:34:21.332070 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529af4d0-f769-47de-af36-9301cad3170e None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:21.332700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529af4d0-f769-47de-af36-9301cad3170e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:21.332700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529af4d0-f769-47de-af36-9301cad3170e None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:21.334058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 555/1112] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:34:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:34:21.341170 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f976749-6c98-4683-b07d-e9e08119e122 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:21.341765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f976749-6c98-4683-b07d-e9e08119e122 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:21.341765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f976749-6c98-4683-b07d-e9e08119e122 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:21.424790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7f976749-6c98-4683-b07d-e9e08119e122 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:21.426070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7f976749-6c98-4683-b07d-e9e08119e122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:21.473023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:21.534106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 558/1113] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:34:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:34:21.685447 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529af4d0-f769-47de-af36-9301cad3170e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:21.686984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-529af4d0-f769-47de-af36-9301cad3170e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:21.856368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529af4d0-f769-47de-af36-9301cad3170e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:21.913391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:21.979645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2175667-6aba-4a98-9ebe-3ab2abb09aa3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:21.980264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2175667-6aba-4a98-9ebe-3ab2abb09aa3 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:21.980542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2175667-6aba-4a98-9ebe-3ab2abb09aa3 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:21.980815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2175667-6aba-4a98-9ebe-3ab2abb09aa3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:21.982424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c2175667-6aba-4a98-9ebe-3ab2abb09aa3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:22.074706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2175667-6aba-4a98-9ebe-3ab2abb09aa3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:22.076299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2175667-6aba-4a98-9ebe-3ab2abb09aa3 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:22.094443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 556/1114] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:23.177518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf4b42-0fda-4e91-ab62-41aac8ebed20 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:23.178071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf4b42-0fda-4e91-ab62-41aac8ebed20 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:23.178071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf4b42-0fda-4e91-ab62-41aac8ebed20 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:23.179553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 559/1115] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:34:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:23.186413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77132f-c769-453f-8fa1-90330def4b93 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:23.186992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77132f-c769-453f-8fa1-90330def4b93 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:23.186992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77132f-c769-453f-8fa1-90330def4b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:23.272473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-cc77132f-c769-453f-8fa1-90330def4b93 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:23.273751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cc77132f-c769-453f-8fa1-90330def4b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:23.276102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:23.383268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 557/1116] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:34:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:23.537712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf4b42-0fda-4e91-ab62-41aac8ebed20 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:23.539277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8bdf4b42-0fda-4e91-ab62-41aac8ebed20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:23.698749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf4b42-0fda-4e91-ab62-41aac8ebed20 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:23.729072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:23.822603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7223b7b-f5dc-42b2-8418-19aa26ec7550 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:23.823220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7223b7b-f5dc-42b2-8418-19aa26ec7550 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:23.823489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7223b7b-f5dc-42b2-8418-19aa26ec7550 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:23.823761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7223b7b-f5dc-42b2-8418-19aa26ec7550 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:23.825389 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c7223b7b-f5dc-42b2-8418-19aa26ec7550 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:23.926460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7223b7b-f5dc-42b2-8418-19aa26ec7550 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:23.928136 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7223b7b-f5dc-42b2-8418-19aa26ec7550 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:23.955351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 560/1117] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:34:26.165359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d42825-2373-4676-9272-880c9cb6d19b None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:26.165996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d42825-2373-4676-9272-880c9cb6d19b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:26.165996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d42825-2373-4676-9272-880c9cb6d19b None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:26.167496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 558/1118] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:26.177453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def90bb-10ef-4618-917f-cd36c469a7cf None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:26.178075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def90bb-10ef-4618-917f-cd36c469a7cf None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:26.178427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def90bb-10ef-4618-917f-cd36c469a7cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:26.255920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:26.304920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6def90bb-10ef-4618-917f-cd36c469a7cf None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:26.307036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6def90bb-10ef-4618-917f-cd36c469a7cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:26.425556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 561/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:26.715309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d42825-2373-4676-9272-880c9cb6d19b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:26.716961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e5d42825-2373-4676-9272-880c9cb6d19b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:26.832378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d42825-2373-4676-9272-880c9cb6d19b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:26.868687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:26.913404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997996c2-eaff-4a34-b505-e30a916d3175 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:26.914011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997996c2-eaff-4a34-b505-e30a916d3175 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:26.914284 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997996c2-eaff-4a34-b505-e30a916d3175 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:26.914581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997996c2-eaff-4a34-b505-e30a916d3175 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:26.916193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-997996c2-eaff-4a34-b505-e30a916d3175 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:27.045528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997996c2-eaff-4a34-b505-e30a916d3175 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:27.047543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997996c2-eaff-4a34-b505-e30a916d3175 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:27.067666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 559/1120] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:29.497096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a1ebd-33f2-426b-a369-eaded2000a8a None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:29.497663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a1ebd-33f2-426b-a369-eaded2000a8a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:29.497663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a1ebd-33f2-426b-a369-eaded2000a8a None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:29.499026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 562/1121] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:29.505722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb3a53-ab15-4bde-8f15-58d0c4816910 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:29.506366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb3a53-ab15-4bde-8f15-58d0c4816910 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:29.506366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb3a53-ab15-4bde-8f15-58d0c4816910 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:29.580167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-87bb3a53-ab15-4bde-8f15-58d0c4816910 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:29.581421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-87bb3a53-ab15-4bde-8f15-58d0c4816910 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:29.595857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:29.692202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 560/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:29.980835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a1ebd-33f2-426b-a369-eaded2000a8a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:29.982628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-424a1ebd-33f2-426b-a369-eaded2000a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:30.108848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a1ebd-33f2-426b-a369-eaded2000a8a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:30.124978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:30.191865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254de8d0-ebad-475d-b192-b9b6d7b36e56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:30.192473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254de8d0-ebad-475d-b192-b9b6d7b36e56 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:30.192754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254de8d0-ebad-475d-b192-b9b6d7b36e56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:30.193026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254de8d0-ebad-475d-b192-b9b6d7b36e56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:30.194702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-254de8d0-ebad-475d-b192-b9b6d7b36e56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:30.318216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254de8d0-ebad-475d-b192-b9b6d7b36e56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:30.319696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254de8d0-ebad-475d-b192-b9b6d7b36e56 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:30.338694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 563/1123] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:33.065589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd1a9c-d0e3-4beb-8bfa-a13822da8f0d None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:33.066176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd1a9c-d0e3-4beb-8bfa-a13822da8f0d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:33.066176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd1a9c-d0e3-4beb-8bfa-a13822da8f0d None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:33.067524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 561/1124] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:34:33.073724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5567818-8e22-4912-912e-10dede395a03 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:33.074252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5567818-8e22-4912-912e-10dede395a03 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:33.074397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5567818-8e22-4912-912e-10dede395a03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:33.130761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:33.161215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a5567818-8e22-4912-912e-10dede395a03 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:33.163259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a5567818-8e22-4912-912e-10dede395a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:33.301510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 564/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:34:33.600835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd1a9c-d0e3-4beb-8bfa-a13822da8f0d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:33.602383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f7fd1a9c-d0e3-4beb-8bfa-a13822da8f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:33.745684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd1a9c-d0e3-4beb-8bfa-a13822da8f0d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:33.818463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:33.858642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fb331f-3ca9-479f-9bbf-fc7d277e00b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:33.860033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fb331f-3ca9-479f-9bbf-fc7d277e00b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:33.860355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fb331f-3ca9-479f-9bbf-fc7d277e00b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:33.860667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fb331f-3ca9-479f-9bbf-fc7d277e00b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:33.862331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-45fb331f-3ca9-479f-9bbf-fc7d277e00b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:33.987896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fb331f-3ca9-479f-9bbf-fc7d277e00b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:33.989371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fb331f-3ca9-479f-9bbf-fc7d277e00b3 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:34.011905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 562/1126] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:34:36.417957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a092da-fe87-4ce5-b6c0-231e114dd6da None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:36.418832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a092da-fe87-4ce5-b6c0-231e114dd6da None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:36.418995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a092da-fe87-4ce5-b6c0-231e114dd6da None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:36.420419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 565/1127] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:36.426651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f929c92-6124-42be-a07d-e92be98e9c49 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:36.427268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f929c92-6124-42be-a07d-e92be98e9c49 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:36.427268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f929c92-6124-42be-a07d-e92be98e9c49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:36.507397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1f929c92-6124-42be-a07d-e92be98e9c49 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:36.508634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1f929c92-6124-42be-a07d-e92be98e9c49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:36.532243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:36.617419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 563/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:36.911047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a092da-fe87-4ce5-b6c0-231e114dd6da None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:36.912819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d7a092da-fe87-4ce5-b6c0-231e114dd6da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:37.028794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a092da-fe87-4ce5-b6c0-231e114dd6da None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:37.052283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:37.112553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaeefc1-8b08-4330-bce4-f416c2887cb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:37.113164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaeefc1-8b08-4330-bce4-f416c2887cb2 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:37.113450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaeefc1-8b08-4330-bce4-f416c2887cb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:37.113721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaeefc1-8b08-4330-bce4-f416c2887cb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:37.115426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bfaeefc1-8b08-4330-bce4-f416c2887cb2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:37.237885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaeefc1-8b08-4330-bce4-f416c2887cb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:37.239365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaeefc1-8b08-4330-bce4-f416c2887cb2 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:37.260004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 566/1129] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:34:39.564781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4510f96f-406d-483b-a8c8-c0b12d09113f None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:39.565347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4510f96f-406d-483b-a8c8-c0b12d09113f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:39.565347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4510f96f-406d-483b-a8c8-c0b12d09113f None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:39.566711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 564/1130] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:34:39.573614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2632a7cd-a74c-4893-b109-a85a83a5b9e2 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:39.574180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2632a7cd-a74c-4893-b109-a85a83a5b9e2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:39.574611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2632a7cd-a74c-4893-b109-a85a83a5b9e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:39.666993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:39.675083 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2632a7cd-a74c-4893-b109-a85a83a5b9e2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:39.677221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2632a7cd-a74c-4893-b109-a85a83a5b9e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:39.811347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 567/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:40.125565 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4510f96f-406d-483b-a8c8-c0b12d09113f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:40.127134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4510f96f-406d-483b-a8c8-c0b12d09113f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:40.252675 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4510f96f-406d-483b-a8c8-c0b12d09113f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:40.323331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:40.351614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec37e685-75d6-4897-a74a-fb01bf6538ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:40.352224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec37e685-75d6-4897-a74a-fb01bf6538ae service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:40.352499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec37e685-75d6-4897-a74a-fb01bf6538ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:40.352772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec37e685-75d6-4897-a74a-fb01bf6538ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:40.354413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ec37e685-75d6-4897-a74a-fb01bf6538ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:40.485516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec37e685-75d6-4897-a74a-fb01bf6538ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:40.486994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec37e685-75d6-4897-a74a-fb01bf6538ae service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:40.508404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 565/1132] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:42.841437 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a6f30-e9c0-4386-938e-56e548416e13 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:42.842016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a6f30-e9c0-4386-938e-56e548416e13 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:42.842016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a6f30-e9c0-4386-938e-56e548416e13 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:42.843347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 568/1133] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:42.849379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38bc923-ce7e-4651-8ad5-49677e8ddbc9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:42.850031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38bc923-ce7e-4651-8ad5-49677e8ddbc9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:42.850194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38bc923-ce7e-4651-8ad5-49677e8ddbc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:42.926041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b38bc923-ce7e-4651-8ad5-49677e8ddbc9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:42.927388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b38bc923-ce7e-4651-8ad5-49677e8ddbc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:42.966513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:43.049899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 566/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:43.346789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4a6f30-e9c0-4386-938e-56e548416e13 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:43.348356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-fc4a6f30-e9c0-4386-938e-56e548416e13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:43.467855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4a6f30-e9c0-4386-938e-56e548416e13 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:43.500939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:43.561421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929151e1-ee20-49c5-b175-d34d8bf1353b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:43.562069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929151e1-ee20-49c5-b175-d34d8bf1353b service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:43.562356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929151e1-ee20-49c5-b175-d34d8bf1353b service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:43.562663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929151e1-ee20-49c5-b175-d34d8bf1353b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:43.564460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-929151e1-ee20-49c5-b175-d34d8bf1353b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:43.706278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929151e1-ee20-49c5-b175-d34d8bf1353b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:43.709267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929151e1-ee20-49c5-b175-d34d8bf1353b service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:43.732616 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 569/1135] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:34:46.053562 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb30e4b6-f3ab-4460-af6e-b88da3fbac27 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:46.054849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb30e4b6-f3ab-4460-af6e-b88da3fbac27 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:46.054849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb30e4b6-f3ab-4460-af6e-b88da3fbac27 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:46.055586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 567/1136] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:46.063357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a262eb4e-77a9-4598-9ed3-869c83cc55b2 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:46.064249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a262eb4e-77a9-4598-9ed3-869c83cc55b2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:46.064249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a262eb4e-77a9-4598-9ed3-869c83cc55b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:46.133422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a262eb4e-77a9-4598-9ed3-869c83cc55b2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:46.134675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a262eb4e-77a9-4598-9ed3-869c83cc55b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:46.143715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:46.240258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 570/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:46.525444 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb30e4b6-f3ab-4460-af6e-b88da3fbac27 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:46.527084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cb30e4b6-f3ab-4460-af6e-b88da3fbac27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:46.642152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb30e4b6-f3ab-4460-af6e-b88da3fbac27 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:46.702964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:46.722193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baacd805-9c2c-4abc-8235-d23d3853ab78 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:46.722846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baacd805-9c2c-4abc-8235-d23d3853ab78 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:46.723130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baacd805-9c2c-4abc-8235-d23d3853ab78 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:46.723398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baacd805-9c2c-4abc-8235-d23d3853ab78 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:46.724995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-baacd805-9c2c-4abc-8235-d23d3853ab78 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:46.843745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baacd805-9c2c-4abc-8235-d23d3853ab78 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:46.845173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baacd805-9c2c-4abc-8235-d23d3853ab78 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:46.866332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 568/1138] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:49.133796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58d841a-c794-4635-8c17-e2ca1f12b552 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:49.134392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58d841a-c794-4635-8c17-e2ca1f12b552 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:49.134392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58d841a-c794-4635-8c17-e2ca1f12b552 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:49.135801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 571/1139] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:49.141904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6b84f8-142b-429e-8ff2-c6b540149227 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:49.142379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6b84f8-142b-429e-8ff2-c6b540149227 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:49.142502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6b84f8-142b-429e-8ff2-c6b540149227 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:49.218276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5f6b84f8-142b-429e-8ff2-c6b540149227 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:49.219460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5f6b84f8-142b-429e-8ff2-c6b540149227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:49.260102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:49.326972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 569/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:34:49.626440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58d841a-c794-4635-8c17-e2ca1f12b552 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:49.628026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b58d841a-c794-4635-8c17-e2ca1f12b552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:49.741044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58d841a-c794-4635-8c17-e2ca1f12b552 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:49.781560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:49.819975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094684fd-f05b-4eb1-bb2a-4ced9ccd08af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:49.820595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094684fd-f05b-4eb1-bb2a-4ced9ccd08af service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:49.820871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094684fd-f05b-4eb1-bb2a-4ced9ccd08af service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:49.821142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094684fd-f05b-4eb1-bb2a-4ced9ccd08af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:49.822807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-094684fd-f05b-4eb1-bb2a-4ced9ccd08af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:49.937191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094684fd-f05b-4eb1-bb2a-4ced9ccd08af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:49.938658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094684fd-f05b-4eb1-bb2a-4ced9ccd08af service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:49.957566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 572/1141] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:52.180687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56552db4-dbaf-4fbe-b585-38d600bf9af8 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:52.181327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56552db4-dbaf-4fbe-b585-38d600bf9af8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:52.181327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56552db4-dbaf-4fbe-b585-38d600bf9af8 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:52.182595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 570/1142] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:52.189150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02891a27-63d5-42db-858e-08d6ba9f37e6 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:52.189710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02891a27-63d5-42db-858e-08d6ba9f37e6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:52.189952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02891a27-63d5-42db-858e-08d6ba9f37e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:52.266996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-02891a27-63d5-42db-858e-08d6ba9f37e6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:52.268293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-02891a27-63d5-42db-858e-08d6ba9f37e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:52.288102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:52.374039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 573/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:52.667986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56552db4-dbaf-4fbe-b585-38d600bf9af8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:52.669643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-56552db4-dbaf-4fbe-b585-38d600bf9af8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:52.787194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56552db4-dbaf-4fbe-b585-38d600bf9af8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:52.860179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:52.867475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea10f13-06be-4e43-98cf-fefc30c4d337 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:52.868091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea10f13-06be-4e43-98cf-fefc30c4d337 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:52.868408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea10f13-06be-4e43-98cf-fefc30c4d337 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:52.868643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea10f13-06be-4e43-98cf-fefc30c4d337 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:52.870282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-bea10f13-06be-4e43-98cf-fefc30c4d337 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:52.981016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea10f13-06be-4e43-98cf-fefc30c4d337 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:52.982464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea10f13-06be-4e43-98cf-fefc30c4d337 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:53.003123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 571/1144] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:34:55.307308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c095d59-b8e4-49e5-a4ae-baf47bcbcab3 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:55.307907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c095d59-b8e4-49e5-a4ae-baf47bcbcab3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:55.307907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c095d59-b8e4-49e5-a4ae-baf47bcbcab3 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:55.309286 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 574/1145] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:34:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:34:55.318524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1303217d-40c9-4936-9255-0b6fafb8168b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:55.319106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1303217d-40c9-4936-9255-0b6fafb8168b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:55.319541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1303217d-40c9-4936-9255-0b6fafb8168b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:55.410206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1303217d-40c9-4936-9255-0b6fafb8168b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:34:55.411490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1303217d-40c9-4936-9255-0b6fafb8168b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:55.497765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:55.583377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 572/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:34:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:34:55.886172 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c095d59-b8e4-49e5-a4ae-baf47bcbcab3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:34:55.887791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2c095d59-b8e4-49e5-a4ae-baf47bcbcab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:56.024512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c095d59-b8e4-49e5-a4ae-baf47bcbcab3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:34:56.083830 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:34:56.113251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4137f4c8-a727-4d11-9c40-2437e5e598de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:34:56.113888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4137f4c8-a727-4d11-9c40-2437e5e598de service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:34:56.114140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4137f4c8-a727-4d11-9c40-2437e5e598de service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:34:56.114410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4137f4c8-a727-4d11-9c40-2437e5e598de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:34:56.116207 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4137f4c8-a727-4d11-9c40-2437e5e598de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:34:56.239083 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4137f4c8-a727-4d11-9c40-2437e5e598de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:34:56.240523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4137f4c8-a727-4d11-9c40-2437e5e598de service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:34:56.260494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 575/1147] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:34:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:35:05.581728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=22 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:06.258008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=66 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:17.002851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb97011-e532-4930-84bc-1042642331d7 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:17.003511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb97011-e532-4930-84bc-1042642331d7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:17.003511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb97011-e532-4930-84bc-1042642331d7 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:17.004750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 573/1148] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:35:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:35:17.010581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee39a9a-0234-4f8e-aa21-503cfbadd64c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:17.011012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee39a9a-0234-4f8e-aa21-503cfbadd64c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:17.011117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee39a9a-0234-4f8e-aa21-503cfbadd64c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:17.080794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7ee39a9a-0234-4f8e-aa21-503cfbadd64c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:35:17.082027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7ee39a9a-0234-4f8e-aa21-503cfbadd64c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:17.181230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 576/1149] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:35:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:35:17.480375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33060be-4cf0-4de2-bd5a-8a7b2dbaf175 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:17.480657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33060be-4cf0-4de2-bd5a-8a7b2dbaf175 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:17.480927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33060be-4cf0-4de2-bd5a-8a7b2dbaf175 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:17.482236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 574/1150] 10.222.0.26 () {54 vars in 962 bytes} [Wed Oct 19 00:35:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:35:17.488044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dae54c-e669-48f2-93b7-77cbc173a10e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:17.488329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dae54c-e669-48f2-93b7-77cbc173a10e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:17.488601 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dae54c-e669-48f2-93b7-77cbc173a10e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:17.569836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-37dae54c-e669-48f2-93b7-77cbc173a10e None None] MFA Rules not processed for user `7e3f6be584e34bfb9e430c7756d071d5`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:35:17.571093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-37dae54c-e669-48f2-93b7-77cbc173a10e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:17.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:17.655592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 577/1151] 10.222.0.26 () {58 vars in 1059 bytes} [Wed Oct 19 00:35:17 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:35:17.663468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33060be-4cf0-4de2-bd5a-8a7b2dbaf175 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:35:17.664963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d33060be-4cf0-4de2-bd5a-8a7b2dbaf175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:17.785533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33060be-4cf0-4de2-bd5a-8a7b2dbaf175 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:35:17.870954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d87cdc-92c0-4c03-9968-691bf966a2a9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e3f6be584e34bfb9e430c7756d071d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:35:17.871773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d87cdc-92c0-4c03-9968-691bf966a2a9 service glance] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:17.872068 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d87cdc-92c0-4c03-9968-691bf966a2a9 service glance] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:17.872347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d87cdc-92c0-4c03-9968-691bf966a2a9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:17.876215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-03d87cdc-92c0-4c03-9968-691bf966a2a9 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:17.941093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:18.010232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d87cdc-92c0-4c03-9968-691bf966a2a9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:35:18.011725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d87cdc-92c0-4c03-9968-691bf966a2a9 service glance] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:35:18.035724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 575/1152] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:35:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:35:20.352986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c76907-2e49-4d82-9e48-efa50de23577 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:20.353551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c76907-2e49-4d82-9e48-efa50de23577 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:20.353661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c76907-2e49-4d82-9e48-efa50de23577 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:20.355242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 578/1153] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:35:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:35:20.361604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676ec716-12ad-451f-8f75-00a09c43f061 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:20.362104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676ec716-12ad-451f-8f75-00a09c43f061 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:20.362235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676ec716-12ad-451f-8f75-00a09c43f061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:20.430821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-676ec716-12ad-451f-8f75-00a09c43f061 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:35:20.431943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-676ec716-12ad-451f-8f75-00a09c43f061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:20.531923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 576/1154] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:35:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:35:20.808033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c76907-2e49-4d82-9e48-efa50de23577 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:35:20.809567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-40c76907-2e49-4d82-9e48-efa50de23577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:20.939742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c76907-2e49-4d82-9e48-efa50de23577 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:35:20.991615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:21.037387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44dd75f-0b2b-4d16-ae6d-ba6a1b844415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:35:21.037990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44dd75f-0b2b-4d16-ae6d-ba6a1b844415 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:21.038296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44dd75f-0b2b-4d16-ae6d-ba6a1b844415 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:21.038599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44dd75f-0b2b-4d16-ae6d-ba6a1b844415 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:21.039236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44dd75f-0b2b-4d16-ae6d-ba6a1b844415 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:35:21.040340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-a44dd75f-0b2b-4d16-ae6d-ba6a1b844415 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:35:21.040340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:35:21.040340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:35:21.041473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44dd75f-0b2b-4d16-ae6d-ba6a1b844415 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:35:21.062277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 579/1155] 10.222.0.26 () {56 vars in 1108 bytes} [Wed Oct 19 00:35:20 2022] GET /identity/v3/projects => generated 2553 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:35:23.343431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e9680e-ccc6-4fc8-9996-9fbb7fe5c8ee None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:23.344010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e9680e-ccc6-4fc8-9996-9fbb7fe5c8ee None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:23.344010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e9680e-ccc6-4fc8-9996-9fbb7fe5c8ee None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:23.345294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 577/1156] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:35:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:35:23.350960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab721c9-bc2b-48d1-922d-626c82e832ed None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:23.351434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab721c9-bc2b-48d1-922d-626c82e832ed None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:23.351530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab721c9-bc2b-48d1-922d-626c82e832ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:23.419390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-cab721c9-bc2b-48d1-922d-626c82e832ed None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:35:23.421376 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-cab721c9-bc2b-48d1-922d-626c82e832ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:23.518128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 580/1157] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:35:23.808423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e9680e-ccc6-4fc8-9996-9fbb7fe5c8ee None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:35:23.810043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a8e9680e-ccc6-4fc8-9996-9fbb7fe5c8ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:23.821876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:23.970695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e9680e-ccc6-4fc8-9996-9fbb7fe5c8ee None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:35:24.057869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:24.098568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcc5d2b-09b1-4219-9554-c1cf21638ef1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:35:24.099207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc5d2b-09b1-4219-9554-c1cf21638ef1 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:24.099467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc5d2b-09b1-4219-9554-c1cf21638ef1 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:24.099731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc5d2b-09b1-4219-9554-c1cf21638ef1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:24.101353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7dcc5d2b-09b1-4219-9554-c1cf21638ef1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:24.196337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcc5d2b-09b1-4219-9554-c1cf21638ef1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:35:24.197949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcc5d2b-09b1-4219-9554-c1cf21638ef1 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:35:24.216665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 578/1158] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:35:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:35:26.607403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6a3587-5438-42ae-b19a-cdd956fe8d07 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:26.608016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6a3587-5438-42ae-b19a-cdd956fe8d07 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:26.608124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6a3587-5438-42ae-b19a-cdd956fe8d07 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:26.609418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 581/1159] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:35:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:35:26.615857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bc43f-abd5-4fb7-8890-771c9907977c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:26.616370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bc43f-abd5-4fb7-8890-771c9907977c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:26.616750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bc43f-abd5-4fb7-8890-771c9907977c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:26.689570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-782bc43f-abd5-4fb7-8890-771c9907977c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:35:26.690859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-782bc43f-abd5-4fb7-8890-771c9907977c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:26.748650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:26.802973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 579/1160] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:35:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:35:27.087886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6a3587-5438-42ae-b19a-cdd956fe8d07 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:35:27.089421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ab6a3587-5438-42ae-b19a-cdd956fe8d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:27.114892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:27.196845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6a3587-5438-42ae-b19a-cdd956fe8d07 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:35:27.272649 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eff671-1700-46a2-aac2-74ee48107bfe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:35:27.273254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eff671-1700-46a2-aac2-74ee48107bfe service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:27.273529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eff671-1700-46a2-aac2-74ee48107bfe service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:27.273795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eff671-1700-46a2-aac2-74ee48107bfe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:27.275442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a4eff671-1700-46a2-aac2-74ee48107bfe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:27.352623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:27.389256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4eff671-1700-46a2-aac2-74ee48107bfe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:35:27.390728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4eff671-1700-46a2-aac2-74ee48107bfe service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:35:27.408878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 582/1161] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:35:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:35:29.636962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f051e-3fff-4837-8fec-666d7b413a81 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:29.637498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f051e-3fff-4837-8fec-666d7b413a81 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:29.637498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f051e-3fff-4837-8fec-666d7b413a81 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:29.639376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 580/1162] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:35:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:35:29.644730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9745cdef-0812-4e81-b86b-b40fb0614016 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:29.645177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9745cdef-0812-4e81-b86b-b40fb0614016 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:29.645281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9745cdef-0812-4e81-b86b-b40fb0614016 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:29.714094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-9745cdef-0812-4e81-b86b-b40fb0614016 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:35:29.715355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9745cdef-0812-4e81-b86b-b40fb0614016 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:29.817677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 583/1163] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:35:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:35:30.108396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338f051e-3fff-4837-8fec-666d7b413a81 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:35:30.109944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-338f051e-3fff-4837-8fec-666d7b413a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:30.224062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338f051e-3fff-4837-8fec-666d7b413a81 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:35:30.299474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:30.301264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce1240-56c9-4895-9624-c840dd5f9e3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:35:30.302095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce1240-56c9-4895-9624-c840dd5f9e3c service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:30.302644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce1240-56c9-4895-9624-c840dd5f9e3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:30.302938 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce1240-56c9-4895-9624-c840dd5f9e3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:30.305011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-91ce1240-56c9-4895-9624-c840dd5f9e3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:30.423406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce1240-56c9-4895-9624-c840dd5f9e3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:35:30.424770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce1240-56c9-4895-9624-c840dd5f9e3c service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:35:30.445010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 581/1164] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:35:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:35:32.754324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef21a6ba-fc0d-4257-b4bd-03d7934724f6 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:32.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef21a6ba-fc0d-4257-b4bd-03d7934724f6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:32.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef21a6ba-fc0d-4257-b4bd-03d7934724f6 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:32.756264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 584/1165] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:35:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:35:32.761602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcff4b9b-5704-4b01-9b60-45d8f35b325b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:32.762052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcff4b9b-5704-4b01-9b60-45d8f35b325b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:32.762207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcff4b9b-5704-4b01-9b60-45d8f35b325b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:32.834108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-fcff4b9b-5704-4b01-9b60-45d8f35b325b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:35:32.835282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fcff4b9b-5704-4b01-9b60-45d8f35b325b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:32.891238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:32.935711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 582/1166] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:35:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:35:33.227544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef21a6ba-fc0d-4257-b4bd-03d7934724f6 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:35:33.229079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ef21a6ba-fc0d-4257-b4bd-03d7934724f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:33.265249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:33.338510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef21a6ba-fc0d-4257-b4bd-03d7934724f6 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:35:33.417885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d2893b-e926-4abf-b77f-ab1e4603624c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:35:33.418508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d2893b-e926-4abf-b77f-ab1e4603624c service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:33.418784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d2893b-e926-4abf-b77f-ab1e4603624c service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:33.419062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d2893b-e926-4abf-b77f-ab1e4603624c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:33.420769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-75d2893b-e926-4abf-b77f-ab1e4603624c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:33.507176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:33.536736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d2893b-e926-4abf-b77f-ab1e4603624c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:35:33.538169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d2893b-e926-4abf-b77f-ab1e4603624c service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:35:33.556512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 585/1167] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:35:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:35:35.984842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c4312-53ff-43a1-861d-109903fbf3b4 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:35.985413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c4312-53ff-43a1-861d-109903fbf3b4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:35.985413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c4312-53ff-43a1-861d-109903fbf3b4 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:35.986861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 583/1168] 10.222.0.26 () {54 vars in 927 bytes} [Wed Oct 19 00:35:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:35:35.992596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71506167-e349-4c23-a417-a041963dd23c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:35.993176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71506167-e349-4c23-a417-a041963dd23c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:35.993176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71506167-e349-4c23-a417-a041963dd23c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:36.066414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-71506167-e349-4c23-a417-a041963dd23c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:35:36.067702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-71506167-e349-4c23-a417-a041963dd23c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:36.170952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 586/1169] 10.222.0.26 () {58 vars in 1024 bytes} [Wed Oct 19 00:35:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:35:36.477323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2c4312-53ff-43a1-861d-109903fbf3b4 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:35:36.478948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7a2c4312-53ff-43a1-861d-109903fbf3b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:36.589927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2c4312-53ff-43a1-861d-109903fbf3b4 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:35:36.666590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9800666-4bbe-4bed-8d4f-530d21cf14b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:35:36.667207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9800666-4bbe-4bed-8d4f-530d21cf14b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:35:36.667489 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9800666-4bbe-4bed-8d4f-530d21cf14b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:35:36.667761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9800666-4bbe-4bed-8d4f-530d21cf14b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:35:36.670140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a9800666-4bbe-4bed-8d4f-530d21cf14b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:35:36.679193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:36.793098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9800666-4bbe-4bed-8d4f-530d21cf14b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:35:36.794790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9800666-4bbe-4bed-8d4f-530d21cf14b4 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:35:36.813767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 584/1170] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:35:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:35:46.168802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=82 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:35:46.811397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=50 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:51.953527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9544c4-ab5b-4ea8-bf32-0691aa3751f1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:51.954170 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9544c4-ab5b-4ea8-bf32-0691aa3751f1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:51.954428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9544c4-ab5b-4ea8-bf32-0691aa3751f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:52.046060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8c9544c4-ab5b-4ea8-bf32-0691aa3751f1 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:36:52.047364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8c9544c4-ab5b-4ea8-bf32-0691aa3751f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:52.159628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 587/1171] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:36:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:36:52.168122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e8bea-4357-4def-a53f-eaa43d9b58e3 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:52.168703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e8bea-4357-4def-a53f-eaa43d9b58e3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:52.168703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e8bea-4357-4def-a53f-eaa43d9b58e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:52.271473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-d64e8bea-4357-4def-a53f-eaa43d9b58e3 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:36:52.272722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d64e8bea-4357-4def-a53f-eaa43d9b58e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:52.387584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 585/1172] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:36:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:36:52.392447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9544c4-ab5b-4ea8-bf32-0691aa3751f1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:52.393689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8c9544c4-ab5b-4ea8-bf32-0691aa3751f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:52.407524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9544c4-ab5b-4ea8-bf32-0691aa3751f1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:52.476742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:52.521224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:52.521838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:52.522110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:52.522378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:52.523049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:36:52.523344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:52.524359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:36:52.524359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:36:52.524359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:36:52.525237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:52.536869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 588/1173] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:36:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:36:52.546745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64e8bea-4357-4def-a53f-eaa43d9b58e3 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:52.558442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64e8bea-4357-4def-a53f-eaa43d9b58e3 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:52.624181 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:52.660295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:52.660919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:52.661192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:52.661461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:52.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:52.663137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING py.warnings [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:36:52.663137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: warnings.warn(msg) Oct 19 00:36:52.663137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: Oct 19 00:36:52.664329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:52.729957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 586/1174] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:36:52 2022] POST /identity/v3/projects => generated 379 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:36:52.734605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:52.736159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:52.774435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6cfec0-3776-4e0f-a47e-dc4eb8a97cfe admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:52.876982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:52.877596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:52.877869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:52.878164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:52.878849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:52.879898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:36:52.879898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:36:52.879898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:36:52.881123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:52.927214 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 589/1175] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:36:52 2022] POST /identity/v3/users => generated 290 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:36:52.931611 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:52.933243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:52.998392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667baca1-48e4-4ac2-9e00-4a8dc442db1a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:53.094246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:53.111389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:53.112411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:53.112879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:53.113335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:53.114096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:53.115795 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:53.130113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 587/1176] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:36:53.134396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:53.136761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:53.149075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac438387-68ce-45f4-b60a-fca25524d387 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:53.176008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:53.250980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c6c8bb-fd12-40b4-9856-61bf321da66d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:53.251538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c6c8bb-fd12-40b4-9856-61bf321da66d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:53.251818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c6c8bb-fd12-40b4-9856-61bf321da66d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:53.252091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c6c8bb-fd12-40b4-9856-61bf321da66d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:53.252749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c6c8bb-fd12-40b4-9856-61bf321da66d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:53.254366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c6c8bb-fd12-40b4-9856-61bf321da66d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:53.268013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 590/1177] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:36:53.273092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:53.274705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:53.310407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aff0ba8-2042-4339-815e-91e3d605777e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:53.415082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97608cc8-0a1c-4e87-8fd5-8911259e7e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:53.415750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97608cc8-0a1c-4e87-8fd5-8911259e7e07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:53.416028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97608cc8-0a1c-4e87-8fd5-8911259e7e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:53.416304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97608cc8-0a1c-4e87-8fd5-8911259e7e07 admin admin] PATH_INFO: `/v3/projects/f1b5c4df58c843c1bb110c959f9a58b5/users/06d47218bdc147edbe7f81c658dfcefc/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:53.442013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97608cc8-0a1c-4e87-8fd5-8911259e7e07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1b5c4df58c843c1bb110c959f9a58b5, user_id=06d47218bdc147edbe7f81c658dfcefc, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:53.444722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97608cc8-0a1c-4e87-8fd5-8911259e7e07 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:53.472013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 588/1178] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:36:53 2022] PUT /identity/v3/projects/f1b5c4df58c843c1bb110c959f9a58b5/users/06d47218bdc147edbe7f81c658dfcefc/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:36:53.483939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ef264-c9e3-4451-a509-f971ee84fed0 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:53.484215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ef264-c9e3-4451-a509-f971ee84fed0 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:53.484484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ef264-c9e3-4451-a509-f971ee84fed0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:53.545023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-760ef264-c9e3-4451-a509-f971ee84fed0 None None] MFA Rules not processed for user `06d47218bdc147edbe7f81c658dfcefc`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:36:53.546200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-760ef264-c9e3-4451-a509-f971ee84fed0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:53.629181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 591/1179] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:36:53 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:36:53.642586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bf06b1-9786-4042-b7bc-bee800ff96a9 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:53.642878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bf06b1-9786-4042-b7bc-bee800ff96a9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:53.643154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bf06b1-9786-4042-b7bc-bee800ff96a9 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:53.644497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 589/1180] 10.222.0.26 () {54 vars in 963 bytes} [Wed Oct 19 00:36:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:36:53.649400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e13a4-b6da-4d52-97ba-4793828a90df None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:53.649689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e13a4-b6da-4d52-97ba-4793828a90df None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:53.649966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e13a4-b6da-4d52-97ba-4793828a90df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:53.687164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:53.734012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ab1e13a4-b6da-4d52-97ba-4793828a90df None None] MFA Rules not processed for user `13270581a86e430886c878ff84302764`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:36:53.735263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ab1e13a4-b6da-4d52-97ba-4793828a90df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:53.814379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 592/1181] 10.222.0.26 () {58 vars in 1060 bytes} [Wed Oct 19 00:36:53 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:36:53.823976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bf06b1-9786-4042-b7bc-bee800ff96a9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:53.825544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-68bf06b1-9786-4042-b7bc-bee800ff96a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:53.879031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:53.935524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bf06b1-9786-4042-b7bc-bee800ff96a9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:54.012236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:54.012853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:54.013222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:54.013407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:54.015065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:54.118679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:54.120104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:54.123629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:54.138863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 590/1182] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:36:53 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:36:54.185977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354221b7-20a6-4fee-870a-d911d2e53f30 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:54.186440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354221b7-20a6-4fee-870a-d911d2e53f30 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:54.186562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354221b7-20a6-4fee-870a-d911d2e53f30 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:54.187819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 593/1183] 10.222.0.26 () {52 vars in 830 bytes} [Wed Oct 19 00:36:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:36:54.191609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:54.192835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:54.299386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233d5f6-017c-4df5-b138-00cc94c08696 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:54.395227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:54.395766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:54.396046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:54.396317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] PATH_INFO: `/v3/users/06d47218bdc147edbe7f81c658dfcefc` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:54.409766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=06d47218bdc147edbe7f81c658dfcefc)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:54.410827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING py.warnings [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:36:54.410827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: warnings.warn(msg) Oct 19 00:36:54.410827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: Oct 19 00:36:54.412052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:54.423142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:54.482259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06d47218bdc147edbe7f81c658dfcefc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:54.498835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06d47218bdc147edbe7f81c658dfcefc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:54.499367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06d47218bdc147edbe7f81c658dfcefc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:54.499923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06d47218bdc147edbe7f81c658dfcefc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:54.510993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06d47218bdc147edbe7f81c658dfcefc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:54.542139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 591/1184] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:36:54 2022] DELETE /identity/v3/users/06d47218bdc147edbe7f81c658dfcefc => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:36:54.546751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95bf88-062c-4e08-b27c-036a6337ae5b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:54.547071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95bf88-062c-4e08-b27c-036a6337ae5b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:54.547351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95bf88-062c-4e08-b27c-036a6337ae5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:54.608392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-0b95bf88-062c-4e08-b27c-036a6337ae5b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:36:54.609607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0b95bf88-062c-4e08-b27c-036a6337ae5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:54.688332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:54.701944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 594/1185] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:36:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:36:54.711821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:54.713076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:54.817860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a7647f-a022-45b0-af34-5ea12d1959d2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:54.875536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:36:54.898832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebb3f05-63af-489e-9011-805350f38306 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:54.899452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb3f05-63af-489e-9011-805350f38306 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:54.899730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb3f05-63af-489e-9011-805350f38306 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:54.900004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb3f05-63af-489e-9011-805350f38306 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:54.901631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-bebb3f05-63af-489e-9011-805350f38306 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:55.025330 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebb3f05-63af-489e-9011-805350f38306 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:55.026853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebb3f05-63af-489e-9011-805350f38306 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:55.047162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 592/1186] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:36:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:36:55.391803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b95bf88-062c-4e08-b27c-036a6337ae5b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:36:55.393062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0b95bf88-062c-4e08-b27c-036a6337ae5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:36:55.405611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b95bf88-062c-4e08-b27c-036a6337ae5b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:36:55.508437 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:36:55.509079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:36:55.509360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:36:55.509634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] PATH_INFO: `/v3/projects/f1b5c4df58c843c1bb110c959f9a58b5` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:36:55.520700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1b5c4df58c843c1bb110c959f9a58b5)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:36:55.521757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:36:55.521757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:36:55.521757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:36:55.522995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:36:55.585959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1b5c4df58c843c1bb110c959f9a58b5', 'request_id': 'req-3b387590-debc-47a3-b9ed-5db422087f18'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:55.601283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1b5c4df58c843c1bb110c959f9a58b5', 'request_id': 'req-3b387590-debc-47a3-b9ed-5db422087f18'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:55.609255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1b5c4df58c843c1bb110c959f9a58b5', 'request_id': 'req-3b387590-debc-47a3-b9ed-5db422087f18'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:55.640277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] The token cache is being invalidate because project f1b5c4df58c843c1bb110c959f9a58b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:36:55.640586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:55.641023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b387590-debc-47a3-b9ed-5db422087f18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:36:55.642502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 595/1187] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:36:55 2022] DELETE /identity/v3/projects/f1b5c4df58c843c1bb110c959f9a58b5 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:37:05.044786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=62 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:37:05.637754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:38:26.018084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f1694-1971-43db-b9cb-1b27b8c2bb23 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:26.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f1694-1971-43db-b9cb-1b27b8c2bb23 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:26.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f1694-1971-43db-b9cb-1b27b8c2bb23 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:26.020146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 593/1188] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:38:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:38:26.025671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356e1691-4df0-43cf-ace6-5558a3a3cd05 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:26.026139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356e1691-4df0-43cf-ace6-5558a3a3cd05 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:26.026245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356e1691-4df0-43cf-ace6-5558a3a3cd05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:26.115730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-356e1691-4df0-43cf-ace6-5558a3a3cd05 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:26.117645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-356e1691-4df0-43cf-ace6-5558a3a3cd05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:26.232794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 596/1189] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:38:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:38:26.388775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62f1694-1971-43db-b9cb-1b27b8c2bb23 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:26.390950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f62f1694-1971-43db-b9cb-1b27b8c2bb23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:26.511987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62f1694-1971-43db-b9cb-1b27b8c2bb23 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:26.517340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2ec4dc-a8ea-4c97-88c7-92202b5faed7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:26.517939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2ec4dc-a8ea-4c97-88c7-92202b5faed7 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:26.518216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2ec4dc-a8ea-4c97-88c7-92202b5faed7 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:26.518504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2ec4dc-a8ea-4c97-88c7-92202b5faed7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:26.526815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2ec4dc-a8ea-4c97-88c7-92202b5faed7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:26.528403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2ec4dc-a8ea-4c97-88c7-92202b5faed7 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:26.537592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 594/1190] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:38:27.528660 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af29b46-bfe7-49e8-9529-21daaab7c152 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:27.529239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af29b46-bfe7-49e8-9529-21daaab7c152 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:27.529239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af29b46-bfe7-49e8-9529-21daaab7c152 None None] PATH_INFO: `/v3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:27.530680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 597/1191] 10.222.0.20 () {54 vars in 925 bytes} [Wed Oct 19 00:38:27 2022] GET /identity/v3 => generated 255 bytes in 6 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:38:27.538075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3e97e-0d6b-4847-aff9-cf0a05bdc77b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:27.539217 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3e97e-0d6b-4847-aff9-cf0a05bdc77b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:27.539814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3e97e-0d6b-4847-aff9-cf0a05bdc77b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:27.605291 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-78d3e97e-0d6b-4847-aff9-cf0a05bdc77b None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:27.606650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-78d3e97e-0d6b-4847-aff9-cf0a05bdc77b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:27.641041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78b3f5-0346-4c5b-ae32-c319ceee6a19 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:27.641432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78b3f5-0346-4c5b-ae32-c319ceee6a19 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:27.641604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78b3f5-0346-4c5b-ae32-c319ceee6a19 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:27.643050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 598/1192] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:38:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 00:38:27.650045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40e7c9c-4b59-4d96-a3b0-e6f784b6250a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:27.650330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40e7c9c-4b59-4d96-a3b0-e6f784b6250a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:27.650645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40e7c9c-4b59-4d96-a3b0-e6f784b6250a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:27.670333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d40e7c9c-4b59-4d96-a3b0-e6f784b6250a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:27.671584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d40e7c9c-4b59-4d96-a3b0-e6f784b6250a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:27.675660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:38:27.679964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 599/1193] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:38:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:38:27.682160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 595/1194] 10.222.0.20 () {58 vars in 1015 bytes} [Wed Oct 19 00:38:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:38:27.833349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40e7c9c-4b59-4d96-a3b0-e6f784b6250a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:27.846392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40e7c9c-4b59-4d96-a3b0-e6f784b6250a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:27.851896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46ee152-f397-4419-bcd0-c7e5c3bea126 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:27.852485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46ee152-f397-4419-bcd0-c7e5c3bea126 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:27.852761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46ee152-f397-4419-bcd0-c7e5c3bea126 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:27.853029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46ee152-f397-4419-bcd0-c7e5c3bea126 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:27.861112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46ee152-f397-4419-bcd0-c7e5c3bea126 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:27.862709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46ee152-f397-4419-bcd0-c7e5c3bea126 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:27.872068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 600/1195] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:38:36.332310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12448ceb-f076-4643-b504-c7a26c6c1e19 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:36.332920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12448ceb-f076-4643-b504-c7a26c6c1e19 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:36.332920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12448ceb-f076-4643-b504-c7a26c6c1e19 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:36.334121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 596/1196] 10.222.0.26 () {54 vars in 961 bytes} [Wed Oct 19 00:38:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:38:36.339773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726f2bd-3f35-4059-80de-4d97f42ddc50 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:36.340202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726f2bd-3f35-4059-80de-4d97f42ddc50 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:36.340318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726f2bd-3f35-4059-80de-4d97f42ddc50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:36.362261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5726f2bd-3f35-4059-80de-4d97f42ddc50 None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:36.363492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5726f2bd-3f35-4059-80de-4d97f42ddc50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:36.371990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 601/1197] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Oct 19 00:38:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:38:36.379696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12448ceb-f076-4643-b504-c7a26c6c1e19 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:36.393176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12448ceb-f076-4643-b504-c7a26c6c1e19 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:36.399010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb2e45c-045e-4125-b91c-f89e589a7ea7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:36.399620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb2e45c-045e-4125-b91c-f89e589a7ea7 service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:36.399924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb2e45c-045e-4125-b91c-f89e589a7ea7 service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:36.400199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb2e45c-045e-4125-b91c-f89e589a7ea7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:36.408871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb2e45c-045e-4125-b91c-f89e589a7ea7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:36.410464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb2e45c-045e-4125-b91c-f89e589a7ea7 service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:36.421380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 597/1198] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:38:39.352082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0179a4-86a4-411a-81a2-66679bd359a4 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:39.352671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0179a4-86a4-411a-81a2-66679bd359a4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:39.352671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0179a4-86a4-411a-81a2-66679bd359a4 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:39.353966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 602/1199] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:38:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:38:39.360272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630774b-8a7a-42a3-a431-ebdd0e890f3c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:39.360708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630774b-8a7a-42a3-a431-ebdd0e890f3c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:39.360819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630774b-8a7a-42a3-a431-ebdd0e890f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:39.382494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e630774b-8a7a-42a3-a431-ebdd0e890f3c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:39.383762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e630774b-8a7a-42a3-a431-ebdd0e890f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:39.391701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 598/1200] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:38:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:38:39.548739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0179a4-86a4-411a-81a2-66679bd359a4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:39.564213 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0179a4-86a4-411a-81a2-66679bd359a4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:39.570583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295cad51-528e-4561-ad10-780205adb0dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:39.571300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295cad51-528e-4561-ad10-780205adb0dc service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:39.571582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295cad51-528e-4561-ad10-780205adb0dc service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:39.571857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295cad51-528e-4561-ad10-780205adb0dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:39.580121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295cad51-528e-4561-ad10-780205adb0dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:39.581706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295cad51-528e-4561-ad10-780205adb0dc service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:39.591933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 603/1201] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:38:40.673420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c9fca8-fd97-472f-bc8f-1ae6fa731cbd None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:40.673956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c9fca8-fd97-472f-bc8f-1ae6fa731cbd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:40.673956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c9fca8-fd97-472f-bc8f-1ae6fa731cbd None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:40.675327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 599/1202] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 00:38:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:38:40.682317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1abb80-740f-4d75-bb12-92921785d57e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:40.682801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1abb80-740f-4d75-bb12-92921785d57e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:40.682904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1abb80-740f-4d75-bb12-92921785d57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:40.702206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6e1abb80-740f-4d75-bb12-92921785d57e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:40.703452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6e1abb80-740f-4d75-bb12-92921785d57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:40.712080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 604/1203] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 00:38:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:38:40.868767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c9fca8-fd97-472f-bc8f-1ae6fa731cbd None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:40.884688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c9fca8-fd97-472f-bc8f-1ae6fa731cbd None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:40.890099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61de2f-7174-43df-b8c5-4937d60b5842 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:40.890723 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61de2f-7174-43df-b8c5-4937d60b5842 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:40.891004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61de2f-7174-43df-b8c5-4937d60b5842 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:40.891274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61de2f-7174-43df-b8c5-4937d60b5842 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:40.900269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61de2f-7174-43df-b8c5-4937d60b5842 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:40.901856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61de2f-7174-43df-b8c5-4937d60b5842 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:40.911563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 600/1204] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:38:44.977706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6bc436-00ff-4551-a0b2-85bf603bcc5b None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:44.978274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6bc436-00ff-4551-a0b2-85bf603bcc5b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:44.978274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6bc436-00ff-4551-a0b2-85bf603bcc5b None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:44.979635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 605/1205] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:38:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:38:44.985012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d56160-0949-4b81-b7f1-3a01b02a9f8f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:44.985446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d56160-0949-4b81-b7f1-3a01b02a9f8f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:44.985562 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d56160-0949-4b81-b7f1-3a01b02a9f8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:45.005821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-23d56160-0949-4b81-b7f1-3a01b02a9f8f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:45.007227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-23d56160-0949-4b81-b7f1-3a01b02a9f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:45.015113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 601/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:38:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:38:45.307489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6bc436-00ff-4551-a0b2-85bf603bcc5b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:45.309194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8a6bc436-00ff-4551-a0b2-85bf603bcc5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:45.336547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:38:45.391615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6bc436-00ff-4551-a0b2-85bf603bcc5b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:45.395281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac700ee0-aa00-4fd5-8bad-993cc260ef29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:45.395892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac700ee0-aa00-4fd5-8bad-993cc260ef29 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:45.396173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac700ee0-aa00-4fd5-8bad-993cc260ef29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:45.396449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac700ee0-aa00-4fd5-8bad-993cc260ef29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:45.406475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac700ee0-aa00-4fd5-8bad-993cc260ef29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:45.407933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac700ee0-aa00-4fd5-8bad-993cc260ef29 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:45.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 606/1207] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:38:47.597978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f887f08-faa5-45c3-adc0-aca931cac9bb None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:47.598581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f887f08-faa5-45c3-adc0-aca931cac9bb None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:47.598581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f887f08-faa5-45c3-adc0-aca931cac9bb None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:47.599962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 602/1208] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:38:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:38:47.606075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5537fc92-f603-407b-af7a-0c2150210c38 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:47.606565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5537fc92-f603-407b-af7a-0c2150210c38 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:47.606683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5537fc92-f603-407b-af7a-0c2150210c38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:47.625856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5537fc92-f603-407b-af7a-0c2150210c38 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:47.627090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5537fc92-f603-407b-af7a-0c2150210c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:47.634851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 607/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:38:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:38:47.929525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f887f08-faa5-45c3-adc0-aca931cac9bb None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:47.940877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f887f08-faa5-45c3-adc0-aca931cac9bb None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:47.945404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7f07e1-a669-4674-898a-4213872a3415 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:47.946018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f07e1-a669-4674-898a-4213872a3415 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:47.946299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f07e1-a669-4674-898a-4213872a3415 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:47.946616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f07e1-a669-4674-898a-4213872a3415 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:47.955945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7f07e1-a669-4674-898a-4213872a3415 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:47.957374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7f07e1-a669-4674-898a-4213872a3415 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:47.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 603/1210] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:38:51.640656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c510990d-d862-477d-904a-a53ffff6eaac None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:51.641209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c510990d-d862-477d-904a-a53ffff6eaac None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:51.641209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c510990d-d862-477d-904a-a53ffff6eaac None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:51.642558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 608/1211] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:38:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:38:51.648872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806776ff-d7e8-47ab-8c87-8f19cdc4e18d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:51.649377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806776ff-d7e8-47ab-8c87-8f19cdc4e18d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:51.649484 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806776ff-d7e8-47ab-8c87-8f19cdc4e18d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:51.669058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-806776ff-d7e8-47ab-8c87-8f19cdc4e18d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:51.670287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-806776ff-d7e8-47ab-8c87-8f19cdc4e18d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:51.678106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 604/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:38:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:38:51.993020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c510990d-d862-477d-904a-a53ffff6eaac None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:51.994726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c510990d-d862-477d-904a-a53ffff6eaac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:52.073614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c510990d-d862-477d-904a-a53ffff6eaac None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:52.077179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4938fe97-31e9-41f1-b681-44ae1390a57c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:52.077775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4938fe97-31e9-41f1-b681-44ae1390a57c service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:52.078051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4938fe97-31e9-41f1-b681-44ae1390a57c service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:52.078318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4938fe97-31e9-41f1-b681-44ae1390a57c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:52.088188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4938fe97-31e9-41f1-b681-44ae1390a57c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:52.089604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4938fe97-31e9-41f1-b681-44ae1390a57c service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:52.099354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 609/1213] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:38:54.258327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd17abb-0a1f-44e8-a122-6be17b74e995 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:54.258890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd17abb-0a1f-44e8-a122-6be17b74e995 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:54.258890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd17abb-0a1f-44e8-a122-6be17b74e995 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:54.260253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 605/1214] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 00:38:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:38:54.265478 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e39c520-286c-4ee7-9af0-d21cd4282b35 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:54.265918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e39c520-286c-4ee7-9af0-d21cd4282b35 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:54.266027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e39c520-286c-4ee7-9af0-d21cd4282b35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:54.286925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8e39c520-286c-4ee7-9af0-d21cd4282b35 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:38:54.288877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8e39c520-286c-4ee7-9af0-d21cd4282b35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:54.297124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 610/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 00:38:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:38:54.623268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd17abb-0a1f-44e8-a122-6be17b74e995 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:38:54.624933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fcd17abb-0a1f-44e8-a122-6be17b74e995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:38:54.638056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd17abb-0a1f-44e8-a122-6be17b74e995 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:38:54.641554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd198a9-1cd3-4228-83e4-6ea295ebbe23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:38:54.642138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd198a9-1cd3-4228-83e4-6ea295ebbe23 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:38:54.642413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd198a9-1cd3-4228-83e4-6ea295ebbe23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:38:54.642777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd198a9-1cd3-4228-83e4-6ea295ebbe23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:38:54.651587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd198a9-1cd3-4228-83e4-6ea295ebbe23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:38:54.652983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd198a9-1cd3-4228-83e4-6ea295ebbe23 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:38:54.662096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 606/1216] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:38:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:39:04.281939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=66 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:39:04.658073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=34 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:20.078293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b23b7-1e7d-4bb5-98d8-850297300215 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.079040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b23b7-1e7d-4bb5-98d8-850297300215 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.079040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b23b7-1e7d-4bb5-98d8-850297300215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.104028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-bb1b23b7-1e7d-4bb5-98d8-850297300215 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.105268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bb1b23b7-1e7d-4bb5-98d8-850297300215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.117662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1fb14-743c-4a7c-a153-22c4ab70b7b9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.118316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1fb14-743c-4a7c-a153-22c4ab70b7b9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.118316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1fb14-743c-4a7c-a153-22c4ab70b7b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.119114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 611/1217] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.130718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4910ee-2a16-4f5f-aef6-9c168e888df9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.131041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4910ee-2a16-4f5f-aef6-9c168e888df9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.131272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4910ee-2a16-4f5f-aef6-9c168e888df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.149955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0ff1fb14-743c-4a7c-a153-22c4ab70b7b9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.151195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0ff1fb14-743c-4a7c-a153-22c4ab70b7b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.163700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2c4910ee-2a16-4f5f-aef6-9c168e888df9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.164936 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2c4910ee-2a16-4f5f-aef6-9c168e888df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.169778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 607/1218] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:40:20.180022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 612/1219] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:40:20.180260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c585a7-749b-4721-baab-be9bda081c7e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.180508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c585a7-749b-4721-baab-be9bda081c7e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.180801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c585a7-749b-4721-baab-be9bda081c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.184865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4910ee-2a16-4f5f-aef6-9c168e888df9 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.198329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4910ee-2a16-4f5f-aef6-9c168e888df9 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.212830 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.213183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e9c585a7-749b-4721-baab-be9bda081c7e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.213723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.214142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.214418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.214575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e9c585a7-749b-4721-baab-be9bda081c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.215072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:20.215310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.217047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.241526 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 608/1220] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:40:20.242572 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7c4cfb-a143-4b29-8c32-3bb2b690eb3c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.242856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7c4cfb-a143-4b29-8c32-3bb2b690eb3c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.243124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7c4cfb-a143-4b29-8c32-3bb2b690eb3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.249530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 613/1221] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:40:20.259625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.282758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65799a75-f12d-4ee4-8de2-8e5e712c664f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.296139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5a7c4cfb-a143-4b29-8c32-3bb2b690eb3c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.296612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbeb76e-3645-4195-b19f-0379dbd7bc3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.297247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbeb76e-3645-4195-b19f-0379dbd7bc3a admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.297373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5a7c4cfb-a143-4b29-8c32-3bb2b690eb3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.297563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbeb76e-3645-4195-b19f-0379dbd7bc3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.297802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbeb76e-3645-4195-b19f-0379dbd7bc3a admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.298458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbeb76e-3645-4195-b19f-0379dbd7bc3a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:20.298797 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbeb76e-3645-4195-b19f-0379dbd7bc3a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.301004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbeb76e-3645-4195-b19f-0379dbd7bc3a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.313222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 609/1222] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.314095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7c4cfb-a143-4b29-8c32-3bb2b690eb3c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.318441 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 614/1223] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:20.324051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd97ba8-feb7-426a-b4a9-d3343c02f31e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.324355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd97ba8-feb7-426a-b4a9-d3343c02f31e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.324640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd97ba8-feb7-426a-b4a9-d3343c02f31e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.331226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7c4cfb-a143-4b29-8c32-3bb2b690eb3c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.341539 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf6411-e512-4e5a-9cc5-3ada830e5270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.342186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf6411-e512-4e5a-9cc5-3ada830e5270 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.342471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf6411-e512-4e5a-9cc5-3ada830e5270 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.342779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf6411-e512-4e5a-9cc5-3ada830e5270 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.343669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adf6411-e512-4e5a-9cc5-3ada830e5270 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.345918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adf6411-e512-4e5a-9cc5-3ada830e5270 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.354177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7bd97ba8-feb7-426a-b4a9-d3343c02f31e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.355436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7bd97ba8-feb7-426a-b4a9-d3343c02f31e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.364810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 615/1224] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.366510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a7537-8779-42f1-8c88-896388c37afe None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.367019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a7537-8779-42f1-8c88-896388c37afe None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.367558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a7537-8779-42f1-8c88-896388c37afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.390845 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 610/1225] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/projects => generated 375 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:20.391502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-df2a7537-8779-42f1-8c88-896388c37afe None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.392340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61244d-93d4-4890-9714-5dc6d8f50ee9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.392635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61244d-93d4-4890-9714-5dc6d8f50ee9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.392987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e61244d-93d4-4890-9714-5dc6d8f50ee9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.394385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-df2a7537-8779-42f1-8c88-896388c37afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.416985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8e61244d-93d4-4890-9714-5dc6d8f50ee9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.418240 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8e61244d-93d4-4890-9714-5dc6d8f50ee9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.458849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 616/1226] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.460108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faafa2c1-4c12-4df2-9ec0-fd4d04575389 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.460422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faafa2c1-4c12-4df2-9ec0-fd4d04575389 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.460723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faafa2c1-4c12-4df2-9ec0-fd4d04575389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.480072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-faafa2c1-4c12-4df2-9ec0-fd4d04575389 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.480274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 611/1227] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.481065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e61244d-93d4-4890-9714-5dc6d8f50ee9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.481363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-faafa2c1-4c12-4df2-9ec0-fd4d04575389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.490208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 617/1228] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.491513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a121f2-03a9-498e-bf2e-4ca44226875b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.491874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a121f2-03a9-498e-bf2e-4ca44226875b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a121f2-03a9-498e-bf2e-4ca44226875b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.496833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e61244d-93d4-4890-9714-5dc6d8f50ee9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.501918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13667b13-ff62-43b9-87c5-105d09d7e34e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.502576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13667b13-ff62-43b9-87c5-105d09d7e34e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.502860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13667b13-ff62-43b9-87c5-105d09d7e34e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.503681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13667b13-ff62-43b9-87c5-105d09d7e34e admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.504350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13667b13-ff62-43b9-87c5-105d09d7e34e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.506300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13667b13-ff62-43b9-87c5-105d09d7e34e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.512699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-f5a121f2-03a9-498e-bf2e-4ca44226875b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.513907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f5a121f2-03a9-498e-bf2e-4ca44226875b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.522740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 618/1229] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.523954 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d39e8b0-121d-428e-babd-c86d75157473 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.524404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d39e8b0-121d-428e-babd-c86d75157473 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.524629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d39e8b0-121d-428e-babd-c86d75157473 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.534889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 612/1230] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:20.536043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13667b13-ff62-43b9-87c5-105d09d7e34e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.544957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-1d39e8b0-121d-428e-babd-c86d75157473 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.546210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1d39e8b0-121d-428e-babd-c86d75157473 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.608876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13667b13-ff62-43b9-87c5-105d09d7e34e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.614570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 619/1231] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.614570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dcc139-bc3c-4474-aa35-e263e2c0402d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.614570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dcc139-bc3c-4474-aa35-e263e2c0402d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.614570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dcc139-bc3c-4474-aa35-e263e2c0402d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.621747 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5195360e-b1ac-4bb2-a8ee-781e53ac1f4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.622637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5195360e-b1ac-4bb2-a8ee-781e53ac1f4e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.623119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5195360e-b1ac-4bb2-a8ee-781e53ac1f4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.623417 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5195360e-b1ac-4bb2-a8ee-781e53ac1f4e admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.624060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5195360e-b1ac-4bb2-a8ee-781e53ac1f4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.625088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING py.warnings [None req-5195360e-b1ac-4bb2-a8ee-781e53ac1f4e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:40:20.625088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: warnings.warn(msg) Oct 19 00:40:20.625088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: Oct 19 00:40:20.626324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5195360e-b1ac-4bb2-a8ee-781e53ac1f4e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.638196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a2dcc139-bc3c-4474-aa35-e263e2c0402d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.639432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a2dcc139-bc3c-4474-aa35-e263e2c0402d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.655773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 620/1232] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.655990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 613/1233] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/users => generated 288 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:20.656116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de9bc5e-65ab-4903-9192-0bffa5de29da None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.656116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de9bc5e-65ab-4903-9192-0bffa5de29da None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.656116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de9bc5e-65ab-4903-9192-0bffa5de29da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.656381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7603002-4093-4879-9b00-a1e98a634845 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.656381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7603002-4093-4879-9b00-a1e98a634845 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.656381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7603002-4093-4879-9b00-a1e98a634845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.670948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-d7603002-4093-4879-9b00-a1e98a634845 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.671484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2de9bc5e-65ab-4903-9192-0bffa5de29da None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.672160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d7603002-4093-4879-9b00-a1e98a634845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.672641 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2de9bc5e-65ab-4903-9192-0bffa5de29da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.680549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 614/1234] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.684697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 621/1235] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.684847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7603002-4093-4879-9b00-a1e98a634845 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.684972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b260f-099a-41a7-a7c2-7838e9d72cf1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.684972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b260f-099a-41a7-a7c2-7838e9d72cf1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.684972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b260f-099a-41a7-a7c2-7838e9d72cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.694797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7603002-4093-4879-9b00-a1e98a634845 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.699642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.700275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.700552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.700830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.701464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:20.701766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.702823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING py.warnings [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:40:20.702823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: warnings.warn(msg) Oct 19 00:40:20.702823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: Oct 19 00:40:20.703707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.704614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-297b260f-099a-41a7-a7c2-7838e9d72cf1 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.706028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-297b260f-099a-41a7-a7c2-7838e9d72cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.711172 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:20.714976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 622/1236] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.715754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b260f-099a-41a7-a7c2-7838e9d72cf1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.720834 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 615/1237] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:20.721848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.732222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573d44af-2bf3-4ca3-9300-8081bc12d784 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.732374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b260f-099a-41a7-a7c2-7838e9d72cf1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.736887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f1e0e7-ad4a-4961-a194-34c20bb19f5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.737041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.738038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f1e0e7-ad4a-4961-a194-34c20bb19f5e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.738371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f1e0e7-ad4a-4961-a194-34c20bb19f5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.738519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.738745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f1e0e7-ad4a-4961-a194-34c20bb19f5e admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.738915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.739255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.739361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f1e0e7-ad4a-4961-a194-34c20bb19f5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.740044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:20.740415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.741274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f1e0e7-ad4a-4961-a194-34c20bb19f5e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.742212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.756289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 616/1238] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:20.757334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.764109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 623/1239] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/users => generated 291 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:20.765734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c846a8e-b3ed-472e-b6e1-2587b2be4486 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.766010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c846a8e-b3ed-472e-b6e1-2587b2be4486 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.766276 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c846a8e-b3ed-472e-b6e1-2587b2be4486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.768230 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499da801-cc67-45a8-8c95-cfbb6f184485 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.773469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca5daa0-dbb5-4faa-a578-7410259c2a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.774150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca5daa0-dbb5-4faa-a578-7410259c2a55 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.774516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca5daa0-dbb5-4faa-a578-7410259c2a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.774862 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca5daa0-dbb5-4faa-a578-7410259c2a55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.775563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca5daa0-dbb5-4faa-a578-7410259c2a55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.776019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:20.777287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca5daa0-dbb5-4faa-a578-7410259c2a55 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.787856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8c846a8e-b3ed-472e-b6e1-2587b2be4486 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:20.788579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 617/1240] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:20.789020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8c846a8e-b3ed-472e-b6e1-2587b2be4486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:20.789493 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca5daa0-dbb5-4faa-a578-7410259c2a55 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.797223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 624/1241] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:20.797882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c846a8e-b3ed-472e-b6e1-2587b2be4486 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.800342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca5daa0-dbb5-4faa-a578-7410259c2a55 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.805382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.805998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.806266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.806578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.807189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c846a8e-b3ed-472e-b6e1-2587b2be4486 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.807292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:20.807513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.809095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.812705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.813334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.813613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.813894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.814584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:20.814890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.817966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.822503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 618/1242] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:20.823477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.834163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8f43c0-5328-42e3-8a6e-bee6894bfb03 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.836184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 625/1243] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:20.836992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.839527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.840262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.840646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.841018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.841766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:20.842471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.844777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.848012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa15914-5fd9-40ba-b1f7-20d4f40af31b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.852849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.853455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.853751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.854021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.854684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.855718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:40:20.855718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:40:20.855718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:40:20.856909 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.864752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 619/1244] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:20.864752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.879155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502b0dcd-f086-4d6c-9dca-ff99b6ebbcb4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.884479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c95438-4ea6-49b3-9ad2-1c825394ec0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.885175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c95438-4ea6-49b3-9ad2-1c825394ec0b admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.885454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c95438-4ea6-49b3-9ad2-1c825394ec0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.885724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c95438-4ea6-49b3-9ad2-1c825394ec0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.886353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c95438-4ea6-49b3-9ad2-1c825394ec0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.887034 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 626/1245] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/projects => generated 405 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:20.887866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.888520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c95438-4ea6-49b3-9ad2-1c825394ec0b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:20.919737 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 620/1246] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:20 2022] POST /identity/v3/projects => generated 391 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:20.923894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c95438-4ea6-49b3-9ad2-1c825394ec0b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:20.953844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e1eb23-7584-4d75-b249-636670e2df27 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.988466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c95438-4ea6-49b3-9ad2-1c825394ec0b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:20.993195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4489c5a4-5e06-440c-abac-48b1c0ef3c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:20.993734 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4489c5a4-5e06-440c-abac-48b1c0ef3c67 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:20.994017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4489c5a4-5e06-440c-abac-48b1c0ef3c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:20.994286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4489c5a4-5e06-440c-abac-48b1c0ef3c67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:20.994994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4489c5a4-5e06-440c-abac-48b1c0ef3c67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:20.996594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4489c5a4-5e06-440c-abac-48b1c0ef3c67 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.007945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 621/1247] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.008937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4489c5a4-5e06-440c-abac-48b1c0ef3c67 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.011944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a808d-426a-442c-b42d-fcb54cbe1018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.012627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a808d-426a-442c-b42d-fcb54cbe1018 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.013053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a808d-426a-442c-b42d-fcb54cbe1018 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.013332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a808d-426a-442c-b42d-fcb54cbe1018 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.013975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a808d-426a-442c-b42d-fcb54cbe1018 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.015640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a808d-426a-442c-b42d-fcb54cbe1018 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.018690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4489c5a4-5e06-440c-abac-48b1c0ef3c67 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.024467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.025156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.025353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.025629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.026274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:21.026720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.026856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 627/1248] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:20 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.027405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a808d-426a-442c-b42d-fcb54cbe1018 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.028339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.037202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a808d-426a-442c-b42d-fcb54cbe1018 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.041233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 622/1249] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:21.042040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.042237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3d45d-dd29-4919-aa4a-fda5fff1789c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.042880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3d45d-dd29-4919-aa4a-fda5fff1789c admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.043163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3d45d-dd29-4919-aa4a-fda5fff1789c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.043436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3d45d-dd29-4919-aa4a-fda5fff1789c admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.044061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b3d45d-dd29-4919-aa4a-fda5fff1789c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.046043 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b3d45d-dd29-4919-aa4a-fda5fff1789c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.054938 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b715ed53-444d-496b-8873-93581fb26a86 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.059824 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b17d67-72a7-435c-bd69-e58f6f547e99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.060435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b17d67-72a7-435c-bd69-e58f6f547e99 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.060708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b17d67-72a7-435c-bd69-e58f6f547e99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.060984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b17d67-72a7-435c-bd69-e58f6f547e99 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.061608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b17d67-72a7-435c-bd69-e58f6f547e99 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.063591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b17d67-72a7-435c-bd69-e58f6f547e99 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.075114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 628/1250] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.076009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3d45d-dd29-4919-aa4a-fda5fff1789c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.096546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 623/1251] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/projects => generated 379 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b17d67-72a7-435c-bd69-e58f6f547e99 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.129271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:21.147988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3d45d-dd29-4919-aa4a-fda5fff1789c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.150994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:21.167008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b17d67-72a7-435c-bd69-e58f6f547e99 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.172115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bf0f02-f2e5-4474-8868-e214b41d5e56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.172762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bf0f02-f2e5-4474-8868-e214b41d5e56 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.173068 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bf0f02-f2e5-4474-8868-e214b41d5e56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.173388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bf0f02-f2e5-4474-8868-e214b41d5e56 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.174019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bf0f02-f2e5-4474-8868-e214b41d5e56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.176277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bf0f02-f2e5-4474-8868-e214b41d5e56 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.198779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 624/1252] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/users => generated 303 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.199481 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bf0f02-f2e5-4474-8868-e214b41d5e56 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.208869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5041e52e-a176-4143-9d7a-48dd70c8e44c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.209524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5041e52e-a176-4143-9d7a-48dd70c8e44c admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.209800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5041e52e-a176-4143-9d7a-48dd70c8e44c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.210069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5041e52e-a176-4143-9d7a-48dd70c8e44c admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.210540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bf0f02-f2e5-4474-8868-e214b41d5e56 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.210723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5041e52e-a176-4143-9d7a-48dd70c8e44c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.212643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5041e52e-a176-4143-9d7a-48dd70c8e44c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.215276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14d690f-6576-4d1e-83f8-c74021aa8335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.215886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d690f-6576-4d1e-83f8-c74021aa8335 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.216152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d690f-6576-4d1e-83f8-c74021aa8335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.216428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d690f-6576-4d1e-83f8-c74021aa8335 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.217047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14d690f-6576-4d1e-83f8-c74021aa8335 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.219121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14d690f-6576-4d1e-83f8-c74021aa8335 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.242495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 629/1253] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/projects => generated 387 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.243253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 625/1254] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/users => generated 296 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.243468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5041e52e-a176-4143-9d7a-48dd70c8e44c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.244005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14d690f-6576-4d1e-83f8-c74021aa8335 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.255040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14d690f-6576-4d1e-83f8-c74021aa8335 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.314152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097a3b81-d793-4c30-8167-41e2cc77614f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.314439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5041e52e-a176-4143-9d7a-48dd70c8e44c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.314811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097a3b81-d793-4c30-8167-41e2cc77614f admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.315091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097a3b81-d793-4c30-8167-41e2cc77614f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.315365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097a3b81-d793-4c30-8167-41e2cc77614f admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.316984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097a3b81-d793-4c30-8167-41e2cc77614f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.318618 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097a3b81-d793-4c30-8167-41e2cc77614f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.319880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1e0dc-1c6b-4a35-9f25-1a21df45a889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.320536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1e0dc-1c6b-4a35-9f25-1a21df45a889 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.320811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1e0dc-1c6b-4a35-9f25-1a21df45a889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.321112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1e0dc-1c6b-4a35-9f25-1a21df45a889 admin admin] PATH_INFO: `/v3/projects/9e9333ca01214933af63450c15a2b51b/users/e9740ef47b0145c18430a28c0e53a0d6/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.328796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 626/1255] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.329594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097a3b81-d793-4c30-8167-41e2cc77614f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.340097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097a3b81-d793-4c30-8167-41e2cc77614f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.340313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd1e0dc-1c6b-4a35-9f25-1a21df45a889 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e9333ca01214933af63450c15a2b51b, user_id=e9740ef47b0145c18430a28c0e53a0d6, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.343088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd1e0dc-1c6b-4a35-9f25-1a21df45a889 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.344829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f123767-8e86-4725-9d4a-df6150e0d851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.345426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f123767-8e86-4725-9d4a-df6150e0d851 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.345972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f123767-8e86-4725-9d4a-df6150e0d851 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.346243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f123767-8e86-4725-9d4a-df6150e0d851 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.347492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f123767-8e86-4725-9d4a-df6150e0d851 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.349358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f123767-8e86-4725-9d4a-df6150e0d851 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.357854 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 630/1256] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:21 2022] PUT /identity/v3/projects/9e9333ca01214933af63450c15a2b51b/users/e9740ef47b0145c18430a28c0e53a0d6/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:21.358761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1e0dc-1c6b-4a35-9f25-1a21df45a889 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.379483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 627/1257] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/projects => generated 379 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.380311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f123767-8e86-4725-9d4a-df6150e0d851 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.436078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1e0dc-1c6b-4a35-9f25-1a21df45a889 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.440843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1644dd30-105d-43e7-b5cc-8eb6161c1878 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.441912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1644dd30-105d-43e7-b5cc-8eb6161c1878 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.442073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f123767-8e86-4725-9d4a-df6150e0d851 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.442201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1644dd30-105d-43e7-b5cc-8eb6161c1878 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.442456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1644dd30-105d-43e7-b5cc-8eb6161c1878 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.443113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1644dd30-105d-43e7-b5cc-8eb6161c1878 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.445041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1644dd30-105d-43e7-b5cc-8eb6161c1878 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.446717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4afe60f-fc88-4117-bd46-61a0e2e6a6b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.447325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4afe60f-fc88-4117-bd46-61a0e2e6a6b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.447602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4afe60f-fc88-4117-bd46-61a0e2e6a6b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.447871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4afe60f-fc88-4117-bd46-61a0e2e6a6b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.448523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4afe60f-fc88-4117-bd46-61a0e2e6a6b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.450459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4afe60f-fc88-4117-bd46-61a0e2e6a6b2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.453428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:21.471709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 631/1258] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/users => generated 293 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.476434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 628/1259] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/users => generated 290 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.476608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1644dd30-105d-43e7-b5cc-8eb6161c1878 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.476728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4afe60f-fc88-4117-bd46-61a0e2e6a6b2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.485677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1644dd30-105d-43e7-b5cc-8eb6161c1878 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.485885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4afe60f-fc88-4117-bd46-61a0e2e6a6b2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.490452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52361c34-f16d-4625-94bf-d4c1cc3bbf40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.490612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3d639f-c0b7-4469-aa3e-929a9ac5b936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.491082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52361c34-f16d-4625-94bf-d4c1cc3bbf40 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.491219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3d639f-c0b7-4469-aa3e-929a9ac5b936 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.491388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52361c34-f16d-4625-94bf-d4c1cc3bbf40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.491506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3d639f-c0b7-4469-aa3e-929a9ac5b936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.491604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52361c34-f16d-4625-94bf-d4c1cc3bbf40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.491692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3d639f-c0b7-4469-aa3e-929a9ac5b936 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.492241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52361c34-f16d-4625-94bf-d4c1cc3bbf40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.492338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3d639f-c0b7-4469-aa3e-929a9ac5b936 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.494047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52361c34-f16d-4625-94bf-d4c1cc3bbf40 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.494282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3d639f-c0b7-4469-aa3e-929a9ac5b936 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.508314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 632/1260] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.509156 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52361c34-f16d-4625-94bf-d4c1cc3bbf40 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.513657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 629/1261] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.514949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3d639f-c0b7-4469-aa3e-929a9ac5b936 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.518948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52361c34-f16d-4625-94bf-d4c1cc3bbf40 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.524761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78615ff6-750f-493d-bb42-2125fff28846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.525357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78615ff6-750f-493d-bb42-2125fff28846 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.525636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78615ff6-750f-493d-bb42-2125fff28846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78615ff6-750f-493d-bb42-2125fff28846 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.526138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3d639f-c0b7-4469-aa3e-929a9ac5b936 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.526569 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78615ff6-750f-493d-bb42-2125fff28846 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.528498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78615ff6-750f-493d-bb42-2125fff28846 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.530765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf705d5-05d1-4953-886a-85af198b0e12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.531409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf705d5-05d1-4953-886a-85af198b0e12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.531691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf705d5-05d1-4953-886a-85af198b0e12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.531957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf705d5-05d1-4953-886a-85af198b0e12 admin admin] PATH_INFO: `/v3/projects/a9ebac69f755473cae560ff06fa55e00/users/2b8ffb23ca6a446493020dfd68fc9eae/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.540679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:21.555691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 633/1262] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/users => generated 294 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.556197 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf705d5-05d1-4953-886a-85af198b0e12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9ebac69f755473cae560ff06fa55e00, user_id=2b8ffb23ca6a446493020dfd68fc9eae, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.556941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8dd7a9-8f01-4871-adfb-dc3f97a5c328 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.557229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8dd7a9-8f01-4871-adfb-dc3f97a5c328 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.557501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8dd7a9-8f01-4871-adfb-dc3f97a5c328 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.559202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf705d5-05d1-4953-886a-85af198b0e12 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.573890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 630/1263] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:21 2022] PUT /identity/v3/projects/a9ebac69f755473cae560ff06fa55e00/users/2b8ffb23ca6a446493020dfd68fc9eae/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:21.575018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf705d5-05d1-4953-886a-85af198b0e12 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.592773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-0e8dd7a9-8f01-4871-adfb-dc3f97a5c328 None None] MFA Rules not processed for user `e9740ef47b0145c18430a28c0e53a0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:21.594003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0e8dd7a9-8f01-4871-adfb-dc3f97a5c328 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:21.649819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf705d5-05d1-4953-886a-85af198b0e12 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.655058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845d30ac-0694-42a2-ba6d-c85273695f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.655663 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845d30ac-0694-42a2-ba6d-c85273695f3a admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.655940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845d30ac-0694-42a2-ba6d-c85273695f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.656216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845d30ac-0694-42a2-ba6d-c85273695f3a admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.656830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845d30ac-0694-42a2-ba6d-c85273695f3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.658768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845d30ac-0694-42a2-ba6d-c85273695f3a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.661743 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 634/1264] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:21.662752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8dd7a9-8f01-4871-adfb-dc3f97a5c328 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.674298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8dd7a9-8f01-4871-adfb-dc3f97a5c328 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.680674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c88b9b3-d3cd-42a8-bc65-f42ccf0659ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.681388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c88b9b3-d3cd-42a8-bc65-f42ccf0659ff admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.681756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c88b9b3-d3cd-42a8-bc65-f42ccf0659ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.682059 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c88b9b3-d3cd-42a8-bc65-f42ccf0659ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.682842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c88b9b3-d3cd-42a8-bc65-f42ccf0659ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.684373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c88b9b3-d3cd-42a8-bc65-f42ccf0659ff admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.692227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 631/1265] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/users => generated 290 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:21.700759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 635/1266] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.701937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c88b9b3-d3cd-42a8-bc65-f42ccf0659ff admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.704953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845d30ac-0694-42a2-ba6d-c85273695f3a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.714281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c88b9b3-d3cd-42a8-bc65-f42ccf0659ff admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.716748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845d30ac-0694-42a2-ba6d-c85273695f3a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.718926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628fa61-e53d-46d3-9771-a8db7d4bef01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.719534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628fa61-e53d-46d3-9771-a8db7d4bef01 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.719810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628fa61-e53d-46d3-9771-a8db7d4bef01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.720082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628fa61-e53d-46d3-9771-a8db7d4bef01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.720973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5628fa61-e53d-46d3-9771-a8db7d4bef01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.722473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7238b1-6d3e-4fd9-8438-468982723001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.723244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5628fa61-e53d-46d3-9771-a8db7d4bef01 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.723369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7238b1-6d3e-4fd9-8438-468982723001 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.723540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7238b1-6d3e-4fd9-8438-468982723001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.723811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7238b1-6d3e-4fd9-8438-468982723001 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.724462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7238b1-6d3e-4fd9-8438-468982723001 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.726021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7238b1-6d3e-4fd9-8438-468982723001 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.733655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 636/1267] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.734561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628fa61-e53d-46d3-9771-a8db7d4bef01 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.738304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 632/1268] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.738304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7238b1-6d3e-4fd9-8438-468982723001 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.745314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628fa61-e53d-46d3-9771-a8db7d4bef01 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.748603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7238b1-6d3e-4fd9-8438-468982723001 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.749922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a49e838-55c7-4a2c-b0c5-2a2ef577d57f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.750641 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a49e838-55c7-4a2c-b0c5-2a2ef577d57f admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.750922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a49e838-55c7-4a2c-b0c5-2a2ef577d57f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.751201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a49e838-55c7-4a2c-b0c5-2a2ef577d57f admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.751863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a49e838-55c7-4a2c-b0c5-2a2ef577d57f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.753512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a49e838-55c7-4a2c-b0c5-2a2ef577d57f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.754080 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5a3300-af8e-4d74-aab1-2bdedd406d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.754787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5a3300-af8e-4d74-aab1-2bdedd406d52 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.755067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5a3300-af8e-4d74-aab1-2bdedd406d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.755337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5a3300-af8e-4d74-aab1-2bdedd406d52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.755964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5a3300-af8e-4d74-aab1-2bdedd406d52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.757569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5a3300-af8e-4d74-aab1-2bdedd406d52 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.764496 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 637/1269] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.765791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae715bc-a46f-4947-8f1f-474755cd2a45 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.766075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae715bc-a46f-4947-8f1f-474755cd2a45 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.766349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae715bc-a46f-4947-8f1f-474755cd2a45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.768639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 633/1270] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.770087 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faa8f4f-4bb9-4f41-bb3c-82cd634a0a4a None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.770362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faa8f4f-4bb9-4f41-bb3c-82cd634a0a4a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.770703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faa8f4f-4bb9-4f41-bb3c-82cd634a0a4a None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.772045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 634/1271] 10.222.0.26 () {54 vars in 961 bytes} [Wed Oct 19 00:40:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:40:21.776373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faa8f4f-4bb9-4f41-bb3c-82cd634a0a4a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.789455 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faa8f4f-4bb9-4f41-bb3c-82cd634a0a4a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.794697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d0611-37d0-4394-97c1-a8b8efb0df1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.795476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d0611-37d0-4394-97c1-a8b8efb0df1c admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.795679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-eae715bc-a46f-4947-8f1f-474755cd2a45 None None] MFA Rules not processed for user `2b8ffb23ca6a446493020dfd68fc9eae`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:21.795795 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d0611-37d0-4394-97c1-a8b8efb0df1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.796036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d0611-37d0-4394-97c1-a8b8efb0df1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.796669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d0611-37d0-4394-97c1-a8b8efb0df1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.796884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-eae715bc-a46f-4947-8f1f-474755cd2a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:21.798269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d0611-37d0-4394-97c1-a8b8efb0df1c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 635/1272] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.809112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d0611-37d0-4394-97c1-a8b8efb0df1c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.819934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d0611-37d0-4394-97c1-a8b8efb0df1c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.824574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ee916c-2d6b-4a9f-8d99-1448a5eec168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.825167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ee916c-2d6b-4a9f-8d99-1448a5eec168 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.825444 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ee916c-2d6b-4a9f-8d99-1448a5eec168 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.825711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ee916c-2d6b-4a9f-8d99-1448a5eec168 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.826364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ee916c-2d6b-4a9f-8d99-1448a5eec168 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.827954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ee916c-2d6b-4a9f-8d99-1448a5eec168 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.838477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 636/1273] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.839278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ee916c-2d6b-4a9f-8d99-1448a5eec168 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.849504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ee916c-2d6b-4a9f-8d99-1448a5eec168 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.854344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2f0589-f5d5-4ce8-8ad8-8d5ec378ba2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.855156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2f0589-f5d5-4ce8-8ad8-8d5ec378ba2b admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.855445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2f0589-f5d5-4ce8-8ad8-8d5ec378ba2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.856184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2f0589-f5d5-4ce8-8ad8-8d5ec378ba2b admin admin] PATH_INFO: `/v3/projects/47e9d7d5512148d5ae8eb9dda5b3657d/users/70a99ac47e8447668deee708fadcbb7d/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.864443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 638/1274] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:21.865278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae715bc-a46f-4947-8f1f-474755cd2a45 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.874055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2f0589-f5d5-4ce8-8ad8-8d5ec378ba2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=47e9d7d5512148d5ae8eb9dda5b3657d, user_id=70a99ac47e8447668deee708fadcbb7d, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.875420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae715bc-a46f-4947-8f1f-474755cd2a45 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.876802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2f0589-f5d5-4ce8-8ad8-8d5ec378ba2b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.880637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1d0c79-dd84-417b-ac4e-001fe3330474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.881238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1d0c79-dd84-417b-ac4e-001fe3330474 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.881621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1d0c79-dd84-417b-ac4e-001fe3330474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.881828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1d0c79-dd84-417b-ac4e-001fe3330474 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.882474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1d0c79-dd84-417b-ac4e-001fe3330474 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.884197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1d0c79-dd84-417b-ac4e-001fe3330474 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.885621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:21.892042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 637/1275] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:21 2022] PUT /identity/v3/projects/47e9d7d5512148d5ae8eb9dda5b3657d/users/70a99ac47e8447668deee708fadcbb7d/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:21.893330 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2f0589-f5d5-4ce8-8ad8-8d5ec378ba2b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.896798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 639/1276] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.898750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1d0c79-dd84-417b-ac4e-001fe3330474 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:21.910877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1d0c79-dd84-417b-ac4e-001fe3330474 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.958906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2f0589-f5d5-4ce8-8ad8-8d5ec378ba2b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:21.963810 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19738bf7-92aa-4be7-b8db-b523013025a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.964470 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19738bf7-92aa-4be7-b8db-b523013025a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.964756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19738bf7-92aa-4be7-b8db-b523013025a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.965031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19738bf7-92aa-4be7-b8db-b523013025a4 admin admin] PATH_INFO: `/v3/projects/c0151d58abd047a692f5cc4e7d05bea5/users/d3f0d615187d4779bdd7da8c9f24c796/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.969735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bc2686-ff98-42e1-b876-87a8f9f00da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:21.969977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bc2686-ff98-42e1-b876-87a8f9f00da0 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.970255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bc2686-ff98-42e1-b876-87a8f9f00da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.970815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bc2686-ff98-42e1-b876-87a8f9f00da0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.971443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bc2686-ff98-42e1-b876-87a8f9f00da0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.973703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bc2686-ff98-42e1-b876-87a8f9f00da0 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:21.980230 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:21.983690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19738bf7-92aa-4be7-b8db-b523013025a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0151d58abd047a692f5cc4e7d05bea5, user_id=d3f0d615187d4779bdd7da8c9f24c796, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:21.983838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 640/1277] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:21.985065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f9ef98-181f-4817-afe3-a87243d4e565 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:21.985281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f9ef98-181f-4817-afe3-a87243d4e565 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:21.985559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f9ef98-181f-4817-afe3-a87243d4e565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:21.987594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19738bf7-92aa-4be7-b8db-b523013025a4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.002366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 638/1278] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:21 2022] PUT /identity/v3/projects/c0151d58abd047a692f5cc4e7d05bea5/users/d3f0d615187d4779bdd7da8c9f24c796/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:22.003520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19738bf7-92aa-4be7-b8db-b523013025a4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.026099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-94f9ef98-181f-4817-afe3-a87243d4e565 None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:22.027349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-94f9ef98-181f-4817-afe3-a87243d4e565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:22.066234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19738bf7-92aa-4be7-b8db-b523013025a4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.070990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252e8fb-5b3f-4786-8ff9-2b8d61c6219d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.071601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252e8fb-5b3f-4786-8ff9-2b8d61c6219d admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.071876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252e8fb-5b3f-4786-8ff9-2b8d61c6219d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.072145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252e8fb-5b3f-4786-8ff9-2b8d61c6219d admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.072950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4252e8fb-5b3f-4786-8ff9-2b8d61c6219d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.074877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4252e8fb-5b3f-4786-8ff9-2b8d61c6219d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.084459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 639/1279] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:22.085231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252e8fb-5b3f-4786-8ff9-2b8d61c6219d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.094919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252e8fb-5b3f-4786-8ff9-2b8d61c6219d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.099868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 641/1280] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Oct 19 00:40:21 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:22.100032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6a1687-dec5-4298-a819-f5f47935d43d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.100426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6a1687-dec5-4298-a819-f5f47935d43d admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.100559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f9ef98-181f-4817-afe3-a87243d4e565 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.100702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6a1687-dec5-4298-a819-f5f47935d43d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.100977 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6a1687-dec5-4298-a819-f5f47935d43d admin admin] PATH_INFO: `/v3/projects/0b35c2038f6b4d0a9012b004f1416c00/users/6f8725c4d05b4dd79b05b43fba15980b/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.111827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f9ef98-181f-4817-afe3-a87243d4e565 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.117283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643720bf-e033-48fc-89dd-406bdbf8343a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.117896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643720bf-e033-48fc-89dd-406bdbf8343a admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.118177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643720bf-e033-48fc-89dd-406bdbf8343a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.118451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643720bf-e033-48fc-89dd-406bdbf8343a admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.119116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643720bf-e033-48fc-89dd-406bdbf8343a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643720bf-e033-48fc-89dd-406bdbf8343a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.123825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6a1687-dec5-4298-a819-f5f47935d43d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b35c2038f6b4d0a9012b004f1416c00, user_id=6f8725c4d05b4dd79b05b43fba15980b, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.126873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6a1687-dec5-4298-a819-f5f47935d43d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.141789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 640/1281] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:22 2022] PUT /identity/v3/projects/0b35c2038f6b4d0a9012b004f1416c00/users/6f8725c4d05b4dd79b05b43fba15980b/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:22.143224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5af423-aac6-4f2b-bb7c-b2851b3a4170 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.143507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5af423-aac6-4f2b-bb7c-b2851b3a4170 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.143887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5af423-aac6-4f2b-bb7c-b2851b3a4170 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.150999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 642/1282] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:22.152371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643720bf-e033-48fc-89dd-406bdbf8343a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.164185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-2a5af423-aac6-4f2b-bb7c-b2851b3a4170 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:22.165413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2a5af423-aac6-4f2b-bb7c-b2851b3a4170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:22.201931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:22.217209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643720bf-e033-48fc-89dd-406bdbf8343a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.223237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f783b40-73f6-4705-b422-e5adf1609954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.224030 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f783b40-73f6-4705-b422-e5adf1609954 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.224451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f783b40-73f6-4705-b422-e5adf1609954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.224848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f783b40-73f6-4705-b422-e5adf1609954 admin admin] PATH_INFO: `/v3/projects/f249d9041c6749f58973a5b0867b07ce/users/8118f0c698aa428a93cb054abac30c2e/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.232979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 641/1283] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:22.232979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5af423-aac6-4f2b-bb7c-b2851b3a4170 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.246829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5af423-aac6-4f2b-bb7c-b2851b3a4170 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.252599 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7992348-7538-4ac5-9f50-1840016a5e04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.254065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7992348-7538-4ac5-9f50-1840016a5e04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.254342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7992348-7538-4ac5-9f50-1840016a5e04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.254496 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f783b40-73f6-4705-b422-e5adf1609954 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f249d9041c6749f58973a5b0867b07ce, user_id=8118f0c698aa428a93cb054abac30c2e, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.254717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7992348-7538-4ac5-9f50-1840016a5e04 admin admin] PATH_INFO: `/v3/projects/37ffcffcae9b41bbb2d13adc49cb99df/users/69fef5650fff42e4ac4ed3ca74d4b123/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.257243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f783b40-73f6-4705-b422-e5adf1609954 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.271917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 643/1284] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:22 2022] PUT /identity/v3/projects/f249d9041c6749f58973a5b0867b07ce/users/8118f0c698aa428a93cb054abac30c2e/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:22.272296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7992348-7538-4ac5-9f50-1840016a5e04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37ffcffcae9b41bbb2d13adc49cb99df, user_id=69fef5650fff42e4ac4ed3ca74d4b123, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.273167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cbb807-f110-4c6e-8f8b-185a79806c41 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.273455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cbb807-f110-4c6e-8f8b-185a79806c41 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.273730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cbb807-f110-4c6e-8f8b-185a79806c41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.275029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7992348-7538-4ac5-9f50-1840016a5e04 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.291114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 642/1285] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:22 2022] PUT /identity/v3/projects/37ffcffcae9b41bbb2d13adc49cb99df/users/69fef5650fff42e4ac4ed3ca74d4b123/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:22.291708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7992348-7538-4ac5-9f50-1840016a5e04 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.296263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-45cbb807-f110-4c6e-8f8b-185a79806c41 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:22.297634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-45cbb807-f110-4c6e-8f8b-185a79806c41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:22.337581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:22.359040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7992348-7538-4ac5-9f50-1840016a5e04 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.359840 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 644/1286] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:22.360669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cbb807-f110-4c6e-8f8b-185a79806c41 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.368165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dbcc68-8994-4edb-9348-f445b2398bb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.369066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dbcc68-8994-4edb-9348-f445b2398bb7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.369209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dbcc68-8994-4edb-9348-f445b2398bb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.369419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dbcc68-8994-4edb-9348-f445b2398bb7 admin admin] PATH_INFO: `/v3/projects/8ed39b7029ea4681aaa9939701c317c1/users/ae9312d545d645c3a44d5ae0fbb99fbe/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.386099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dbcc68-8994-4edb-9348-f445b2398bb7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ed39b7029ea4681aaa9939701c317c1, user_id=ae9312d545d645c3a44d5ae0fbb99fbe, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.389228 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dbcc68-8994-4edb-9348-f445b2398bb7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.403079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 643/1287] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:22 2022] PUT /identity/v3/projects/8ed39b7029ea4681aaa9939701c317c1/users/ae9312d545d645c3a44d5ae0fbb99fbe/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:22.407849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541f55ec-bb88-468f-98d1-921368a7576d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.408545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541f55ec-bb88-468f-98d1-921368a7576d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.408846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541f55ec-bb88-468f-98d1-921368a7576d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.429910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-541f55ec-bb88-468f-98d1-921368a7576d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:22.431253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-541f55ec-bb88-468f-98d1-921368a7576d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:22.437606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cbb807-f110-4c6e-8f8b-185a79806c41 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.494998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 644/1288] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:22.495838 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541f55ec-bb88-468f-98d1-921368a7576d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.503357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d71e6c-f114-41e7-839d-f48ef1a1dea6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.504339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d71e6c-f114-41e7-839d-f48ef1a1dea6 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.504614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d71e6c-f114-41e7-839d-f48ef1a1dea6 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.504886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d71e6c-f114-41e7-839d-f48ef1a1dea6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.505970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541f55ec-bb88-468f-98d1-921368a7576d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.510342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:22.510668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d323916e-aa86-4a6b-9f35-4b9892f66ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.511278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d323916e-aa86-4a6b-9f35-4b9892f66ed5 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.511559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d323916e-aa86-4a6b-9f35-4b9892f66ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.511831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d323916e-aa86-4a6b-9f35-4b9892f66ed5 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.512466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d323916e-aa86-4a6b-9f35-4b9892f66ed5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.514779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d323916e-aa86-4a6b-9f35-4b9892f66ed5 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.537828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 645/1289] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/users => generated 297 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:22.538944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d323916e-aa86-4a6b-9f35-4b9892f66ed5 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.573468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d71e6c-f114-41e7-839d-f48ef1a1dea6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.575109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d71e6c-f114-41e7-839d-f48ef1a1dea6 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.584694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 645/1290] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:22.585924 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19aa9a1-cc17-4b66-95ca-748d7239d9d8 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.586211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19aa9a1-cc17-4b66-95ca-748d7239d9d8 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.586510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19aa9a1-cc17-4b66-95ca-748d7239d9d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.600205 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d323916e-aa86-4a6b-9f35-4b9892f66ed5 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.604022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.604621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.604901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.605172 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.606540 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a19aa9a1-cc17-4b66-95ca-748d7239d9d8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:22.607804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a19aa9a1-cc17-4b66-95ca-748d7239d9d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:22.615552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.619186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 646/1291] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:22.619344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.619444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eba2bb7-d301-4b74-af58-228ea8251023 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.619444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eba2bb7-d301-4b74-af58-228ea8251023 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.619444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eba2bb7-d301-4b74-af58-228ea8251023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.627425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 646/1292] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:22.628921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.630307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:22.638317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-0eba2bb7-d301-4b74-af58-228ea8251023 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:22.639550 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0eba2bb7-d301-4b74-af58-228ea8251023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:22.641546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ce1358-9d21-4add-9d4e-a3f82216fd89 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.646113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d6fb90-048e-4ed3-9797-bbfd44e67c8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.646867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d6fb90-048e-4ed3-9797-bbfd44e67c8b service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.647241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d6fb90-048e-4ed3-9797-bbfd44e67c8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.647630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d6fb90-048e-4ed3-9797-bbfd44e67c8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.648683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 647/1293] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:22.652998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0b212-99c2-4f11-90b4-e5f6cf831e19 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.653280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0b212-99c2-4f11-90b4-e5f6cf831e19 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.653554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0b212-99c2-4f11-90b4-e5f6cf831e19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.666825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d6fb90-048e-4ed3-9797-bbfd44e67c8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.670109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d6fb90-048e-4ed3-9797-bbfd44e67c8b service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.687308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6ed0b212-99c2-4f11-90b4-e5f6cf831e19 None None] MFA Rules not processed for user `ae9312d545d645c3a44d5ae0fbb99fbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:22.689271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6ed0b212-99c2-4f11-90b4-e5f6cf831e19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:22.698159 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 647/1294] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:22.699292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d6fb90-048e-4ed3-9797-bbfd44e67c8b service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.719753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d6fb90-048e-4ed3-9797-bbfd44e67c8b service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.725469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce050032-e619-4ad8-ba56-d74c6bdd9ff0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.726851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce050032-e619-4ad8-ba56-d74c6bdd9ff0 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.727355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce050032-e619-4ad8-ba56-d74c6bdd9ff0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.727846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce050032-e619-4ad8-ba56-d74c6bdd9ff0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.741958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce050032-e619-4ad8-ba56-d74c6bdd9ff0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.744362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce050032-e619-4ad8-ba56-d74c6bdd9ff0 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.759672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 648/1295] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:22.762286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce050032-e619-4ad8-ba56-d74c6bdd9ff0 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.776333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce050032-e619-4ad8-ba56-d74c6bdd9ff0 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.791772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2c9dec-5b19-4a74-96ae-0d896d3bf09c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.792925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2c9dec-5b19-4a74-96ae-0d896d3bf09c admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.793415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2c9dec-5b19-4a74-96ae-0d896d3bf09c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.793913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2c9dec-5b19-4a74-96ae-0d896d3bf09c admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.795208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2c9dec-5b19-4a74-96ae-0d896d3bf09c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.797126 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2c9dec-5b19-4a74-96ae-0d896d3bf09c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.811568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 649/1296] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:22.811568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2c9dec-5b19-4a74-96ae-0d896d3bf09c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.814940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 648/1297] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:22.814940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed0b212-99c2-4f11-90b4-e5f6cf831e19 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.833232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed0b212-99c2-4f11-90b4-e5f6cf831e19 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.838514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf837b8b-b2c1-4947-b0e3-d5ca1198f33e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.839282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf837b8b-b2c1-4947-b0e3-d5ca1198f33e service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.839645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf837b8b-b2c1-4947-b0e3-d5ca1198f33e service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.840021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf837b8b-b2c1-4947-b0e3-d5ca1198f33e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.843255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2c9dec-5b19-4a74-96ae-0d896d3bf09c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.850059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75cf0f0-17fd-42f7-8dd5-211fa5e53cc7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.850868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75cf0f0-17fd-42f7-8dd5-211fa5e53cc7 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.851445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75cf0f0-17fd-42f7-8dd5-211fa5e53cc7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.851855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75cf0f0-17fd-42f7-8dd5-211fa5e53cc7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.862176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf837b8b-b2c1-4947-b0e3-d5ca1198f33e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.864917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf837b8b-b2c1-4947-b0e3-d5ca1198f33e service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.866037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75cf0f0-17fd-42f7-8dd5-211fa5e53cc7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.868007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75cf0f0-17fd-42f7-8dd5-211fa5e53cc7 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.873124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:22.887386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 650/1298] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:22.888327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75cf0f0-17fd-42f7-8dd5-211fa5e53cc7 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.893898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 649/1299] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:22.895194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf837b8b-b2c1-4947-b0e3-d5ca1198f33e service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:22.902104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75cf0f0-17fd-42f7-8dd5-211fa5e53cc7 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.908604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c9cdb-0043-49dd-a96f-6b723a965012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.909564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c9cdb-0043-49dd-a96f-6b723a965012 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.910046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c9cdb-0043-49dd-a96f-6b723a965012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.910379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362c9cdb-0043-49dd-a96f-6b723a965012 admin admin] PATH_INFO: `/v3/projects/28a5bc93e44c492e8c8c13426423b02a/users/ffbead23e0104ab2a2f4a9373b6c1ebc/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.913547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf837b8b-b2c1-4947-b0e3-d5ca1198f33e service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:22.930732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a6203-7cad-4cbb-b505-0486b47ce0dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:22.931359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a6203-7cad-4cbb-b505-0486b47ce0dd admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:22.931647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a6203-7cad-4cbb-b505-0486b47ce0dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:22.931916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a6203-7cad-4cbb-b505-0486b47ce0dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:22.932576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671a6203-7cad-4cbb-b505-0486b47ce0dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.934531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671a6203-7cad-4cbb-b505-0486b47ce0dd admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.936845 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362c9cdb-0043-49dd-a96f-6b723a965012 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28a5bc93e44c492e8c8c13426423b02a, user_id=ffbead23e0104ab2a2f4a9373b6c1ebc, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:22.941653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362c9cdb-0043-49dd-a96f-6b723a965012 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:22.961462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 651/1300] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:22 2022] PUT /identity/v3/projects/28a5bc93e44c492e8c8c13426423b02a/users/ffbead23e0104ab2a2f4a9373b6c1ebc/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:22.965475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c9cdb-0043-49dd-a96f-6b723a965012 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.007732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 650/1301] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:22 2022] POST /identity/v3/projects => generated 381 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:23.011698 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a6203-7cad-4cbb-b505-0486b47ce0dd admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.091026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:23.124911 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362c9cdb-0043-49dd-a96f-6b723a965012 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:23.131893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd896376-28e3-4326-a6c6-f19b77dbb3f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:23.132817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd896376-28e3-4326-a6c6-f19b77dbb3f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.133341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd896376-28e3-4326-a6c6-f19b77dbb3f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.133505 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a6203-7cad-4cbb-b505-0486b47ce0dd admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:23.133927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd896376-28e3-4326-a6c6-f19b77dbb3f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.134852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd896376-28e3-4326-a6c6-f19b77dbb3f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:23.136840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd896376-28e3-4326-a6c6-f19b77dbb3f1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:23.144938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04d76ac-c7f0-4476-a463-08f4eea63cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:23.148095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04d76ac-c7f0-4476-a463-08f4eea63cac admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.148537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04d76ac-c7f0-4476-a463-08f4eea63cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.153181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04d76ac-c7f0-4476-a463-08f4eea63cac admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.154114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04d76ac-c7f0-4476-a463-08f4eea63cac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:23.161698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 652/1302] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:22 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:23.163888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a7b74-0fd4-433c-b39b-b209bef114ee None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.164989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a7b74-0fd4-433c-b39b-b209bef114ee None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.166598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a7b74-0fd4-433c-b39b-b209bef114ee None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.169684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 653/1303] 10.222.0.26 () {54 vars in 961 bytes} [Wed Oct 19 00:40:23 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:40:23.171816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a7b74-0fd4-433c-b39b-b209bef114ee None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.174121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04d76ac-c7f0-4476-a463-08f4eea63cac admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:23.199820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a7b74-0fd4-433c-b39b-b209bef114ee None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:23.207409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b072cae-e834-43fb-9f0f-315636d2e30a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:23.209052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b072cae-e834-43fb-9f0f-315636d2e30a admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.209516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b072cae-e834-43fb-9f0f-315636d2e30a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.209751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b072cae-e834-43fb-9f0f-315636d2e30a admin admin] PATH_INFO: `/v3/projects/28a5bc93e44c492e8c8c13426423b02a/users/ffbead23e0104ab2a2f4a9373b6c1ebc/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.213204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b072cae-e834-43fb-9f0f-315636d2e30a admin admin] RBAC: Authorizing `identity:create_grant(project_id=28a5bc93e44c492e8c8c13426423b02a, user_id=ffbead23e0104ab2a2f4a9373b6c1ebc, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:23.217422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b072cae-e834-43fb-9f0f-315636d2e30a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:23.228666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 651/1304] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:23 2022] POST /identity/v3/users => generated 297 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:23.230250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bc5bcc-ef14-41c4-9f3c-c7ecff149b45 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.230692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bc5bcc-ef14-41c4-9f3c-c7ecff149b45 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.231038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bc5bcc-ef14-41c4-9f3c-c7ecff149b45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.235128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 654/1305] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:23 2022] PUT /identity/v3/projects/28a5bc93e44c492e8c8c13426423b02a/users/ffbead23e0104ab2a2f4a9373b6c1ebc/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:23.236406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b072cae-e834-43fb-9f0f-315636d2e30a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.262601 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-86bc5bcc-ef14-41c4-9f3c-c7ecff149b45 None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:23.263837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-86bc5bcc-ef14-41c4-9f3c-c7ecff149b45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:23.338415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b072cae-e834-43fb-9f0f-315636d2e30a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:23.349793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5fe309-1ca5-4f87-8b0a-853f8d6561be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:23.351042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5fe309-1ca5-4f87-8b0a-853f8d6561be admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.351522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5fe309-1ca5-4f87-8b0a-853f8d6561be admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.351946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5fe309-1ca5-4f87-8b0a-853f8d6561be admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.352905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5fe309-1ca5-4f87-8b0a-853f8d6561be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:23.355711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5fe309-1ca5-4f87-8b0a-853f8d6561be admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:23.362379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 652/1306] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Oct 19 00:40:23 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:23.363858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ac8342-2635-4b58-82c9-f75805d869a1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.364191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ac8342-2635-4b58-82c9-f75805d869a1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.364457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ac8342-2635-4b58-82c9-f75805d869a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.372626 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 655/1307] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:23 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:23.374758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee6082-3042-4405-8efb-97a79f5beca8 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.375703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee6082-3042-4405-8efb-97a79f5beca8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.376144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee6082-3042-4405-8efb-97a79f5beca8 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.378101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 656/1308] 10.222.0.26 () {54 vars in 912 bytes} [Wed Oct 19 00:40:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:40:23.379260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcee6082-3042-4405-8efb-97a79f5beca8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.397839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcee6082-3042-4405-8efb-97a79f5beca8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:23.404084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c3ac8342-2635-4b58-82c9-f75805d869a1 None None] MFA Rules not processed for user `ffbead23e0104ab2a2f4a9373b6c1ebc`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:23.405482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c3ac8342-2635-4b58-82c9-f75805d869a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:23.405624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd80bb4-1c4b-4029-bbf9-9361a1c34a12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:23.406476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd80bb4-1c4b-4029-bbf9-9361a1c34a12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.406959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd80bb4-1c4b-4029-bbf9-9361a1c34a12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.407413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd80bb4-1c4b-4029-bbf9-9361a1c34a12 admin admin] PATH_INFO: `/v3/projects/61b20b3d0794441ca2f960285a767476/users/4747e094d75d4d7b9cc498bf26ea1727/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.439403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd80bb4-1c4b-4029-bbf9-9361a1c34a12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61b20b3d0794441ca2f960285a767476, user_id=4747e094d75d4d7b9cc498bf26ea1727, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:23.454411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd80bb4-1c4b-4029-bbf9-9361a1c34a12 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:23.484472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 657/1309] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:23 2022] PUT /identity/v3/projects/61b20b3d0794441ca2f960285a767476/users/4747e094d75d4d7b9cc498bf26ea1727/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:23.488129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e60baca-04a6-4239-bf15-487c69a307d0 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.488129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e60baca-04a6-4239-bf15-487c69a307d0 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.488371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e60baca-04a6-4239-bf15-487c69a307d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.527237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 653/1310] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:23 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:23.529241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ac8342-2635-4b58-82c9-f75805d869a1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.540085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8e60baca-04a6-4239-bf15-487c69a307d0 None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:23.543353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8e60baca-04a6-4239-bf15-487c69a307d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:23.544426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ac8342-2635-4b58-82c9-f75805d869a1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:23.604386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:23.628095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:23.640920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5faca75-82de-44d7-a2a8-72a2b795b336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:23.642090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5faca75-82de-44d7-a2a8-72a2b795b336 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.642834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5faca75-82de-44d7-a2a8-72a2b795b336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.643216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5faca75-82de-44d7-a2a8-72a2b795b336 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.644670 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5faca75-82de-44d7-a2a8-72a2b795b336 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:23.646742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5faca75-82de-44d7-a2a8-72a2b795b336 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:23.670155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 654/1311] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:23 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:23.670155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5faca75-82de-44d7-a2a8-72a2b795b336 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.702322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 658/1312] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:40:23 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:23.704744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e60baca-04a6-4239-bf15-487c69a307d0 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.748602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e60baca-04a6-4239-bf15-487c69a307d0 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:23.763476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584585e3-85f4-4b6f-b59c-775c05b0f5ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:23.770764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584585e3-85f4-4b6f-b59c-775c05b0f5ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.771276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584585e3-85f4-4b6f-b59c-775c05b0f5ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.771657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584585e3-85f4-4b6f-b59c-775c05b0f5ca admin admin] PATH_INFO: `/v3/projects/61b20b3d0794441ca2f960285a767476/users/4747e094d75d4d7b9cc498bf26ea1727/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.776383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584585e3-85f4-4b6f-b59c-775c05b0f5ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=61b20b3d0794441ca2f960285a767476, user_id=4747e094d75d4d7b9cc498bf26ea1727, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:23.783323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5faca75-82de-44d7-a2a8-72a2b795b336 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:23.783608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584585e3-85f4-4b6f-b59c-775c05b0f5ca admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:23.791683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1bfa17-0c6b-4a47-8432-eeabef2e1224 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:23.793622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1bfa17-0c6b-4a47-8432-eeabef2e1224 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.795055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1bfa17-0c6b-4a47-8432-eeabef2e1224 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.796428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1bfa17-0c6b-4a47-8432-eeabef2e1224 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.825684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 659/1313] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:23 2022] PUT /identity/v3/projects/61b20b3d0794441ca2f960285a767476/users/4747e094d75d4d7b9cc498bf26ea1727/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:23.825684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584585e3-85f4-4b6f-b59c-775c05b0f5ca admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:23.891340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1bfa17-0c6b-4a47-8432-eeabef2e1224 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:23.893434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1bfa17-0c6b-4a47-8432-eeabef2e1224 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:23.909228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 655/1314] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:23 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:23.911295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ea5ca8-cd3f-425e-950a-24b24647b902 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:23.911847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ea5ca8-cd3f-425e-950a-24b24647b902 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:23.913223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ea5ca8-cd3f-425e-950a-24b24647b902 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:23.988424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-35ea5ca8-cd3f-425e-950a-24b24647b902 None None] MFA Rules not processed for user `4747e094d75d4d7b9cc498bf26ea1727`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:23.992615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-35ea5ca8-cd3f-425e-950a-24b24647b902 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:24.014952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584585e3-85f4-4b6f-b59c-775c05b0f5ca admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:24.027283 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cdc9a-66f1-47e6-92ac-0dfd4e3927ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:24.027975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cdc9a-66f1-47e6-92ac-0dfd4e3927ca service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:24.028253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cdc9a-66f1-47e6-92ac-0dfd4e3927ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:24.028541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cdc9a-66f1-47e6-92ac-0dfd4e3927ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:24.177035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3cdc9a-66f1-47e6-92ac-0dfd4e3927ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:24.179622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3cdc9a-66f1-47e6-92ac-0dfd4e3927ca service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:24.187117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 656/1315] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:23 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:24.187117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ea5ca8-cd3f-425e-950a-24b24647b902 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:24.200178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ea5ca8-cd3f-425e-950a-24b24647b902 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:24.207020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3893b948-d25e-4266-8ea6-2eb940b6efeb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:24.207894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893b948-d25e-4266-8ea6-2eb940b6efeb service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:24.208319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893b948-d25e-4266-8ea6-2eb940b6efeb service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:24.209005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893b948-d25e-4266-8ea6-2eb940b6efeb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:24.214364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 660/1316] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:24.247470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:24.293564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cdc9a-66f1-47e6-92ac-0dfd4e3927ca service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:24.317785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cdc9a-66f1-47e6-92ac-0dfd4e3927ca service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:24.335287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d5308c-888b-482b-8cda-b300b7d47702 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:24.336306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d5308c-888b-482b-8cda-b300b7d47702 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:24.337338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d5308c-888b-482b-8cda-b300b7d47702 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:24.337930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d5308c-888b-482b-8cda-b300b7d47702 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:24.371604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d5308c-888b-482b-8cda-b300b7d47702 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:24.373128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d5308c-888b-482b-8cda-b300b7d47702 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:24.388572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3893b948-d25e-4266-8ea6-2eb940b6efeb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:24.390516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3893b948-d25e-4266-8ea6-2eb940b6efeb service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:24.418923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 661/1317] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:24 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:24.420537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d6286-850a-49e9-a9f0-9c60d8d59b00 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:24.422176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d6286-850a-49e9-a9f0-9c60d8d59b00 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:24.422804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d6286-850a-49e9-a9f0-9c60d8d59b00 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:24.440503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 657/1318] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:24 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:24.442406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030af068-7793-4283-b8bb-22ef0b0adcf1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:24.443885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 662/1319] 10.222.0.26 () {54 vars in 912 bytes} [Wed Oct 19 00:40:24 2022] GET /identity => generated 270 bytes in 24 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:40:24.447117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030af068-7793-4283-b8bb-22ef0b0adcf1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:24.451435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030af068-7793-4283-b8bb-22ef0b0adcf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:24.493158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-030af068-7793-4283-b8bb-22ef0b0adcf1 None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:24.497109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-030af068-7793-4283-b8bb-22ef0b0adcf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:24.535391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 658/1320] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:40:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:24.630453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e427b93-13a5-4303-bfc4-314ac7deb980 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:24.630921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e427b93-13a5-4303-bfc4-314ac7deb980 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:24.631293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e427b93-13a5-4303-bfc4-314ac7deb980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:24.634241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030af068-7793-4283-b8bb-22ef0b0adcf1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:24.652002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030af068-7793-4283-b8bb-22ef0b0adcf1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:24.663391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe134a3-1c66-4d82-9c3f-9fd76e6c2bce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:24.664049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe134a3-1c66-4d82-9c3f-9fd76e6c2bce service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:24.664486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe134a3-1c66-4d82-9c3f-9fd76e6c2bce service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:24.664853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe134a3-1c66-4d82-9c3f-9fd76e6c2bce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:24.692578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe134a3-1c66-4d82-9c3f-9fd76e6c2bce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:24.694360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe134a3-1c66-4d82-9c3f-9fd76e6c2bce service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:24.704682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1e427b93-13a5-4303-bfc4-314ac7deb980 None None] MFA Rules not processed for user `70a99ac47e8447668deee708fadcbb7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:24.705925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1e427b93-13a5-4303-bfc4-314ac7deb980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:24.720275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 659/1321] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:40:24.776411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:24.915546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 663/1322] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:24 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:24.921297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe134a3-1c66-4d82-9c3f-9fd76e6c2bce service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:25.036149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e427b93-13a5-4303-bfc4-314ac7deb980 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:25.158119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe134a3-1c66-4d82-9c3f-9fd76e6c2bce service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:25.169620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97896d-d82e-4847-8124-d5018a13ae9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:25.171858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97896d-d82e-4847-8124-d5018a13ae9a admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97896d-d82e-4847-8124-d5018a13ae9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.172414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97896d-d82e-4847-8124-d5018a13ae9a admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.173075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef97896d-d82e-4847-8124-d5018a13ae9a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:25.185272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e427b93-13a5-4303-bfc4-314ac7deb980 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:25.186052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef97896d-d82e-4847-8124-d5018a13ae9a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:25.192455 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:25.193190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.194571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.195309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] PATH_INFO: `/v3/users/e9740ef47b0145c18430a28c0e53a0d6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.197075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9740ef47b0145c18430a28c0e53a0d6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:25.200728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:25.253842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 660/1323] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:24 2022] POST /identity/v3/projects => generated 403 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:25.259038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97896d-d82e-4847-8124-d5018a13ae9a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:25.308117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9740ef47b0145c18430a28c0e53a0d6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:25.326127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9740ef47b0145c18430a28c0e53a0d6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:25.328010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9740ef47b0145c18430a28c0e53a0d6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:25.329347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9740ef47b0145c18430a28c0e53a0d6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:25.345772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-63f96e41-d26d-4d70-97f6-60bdcd3054f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9740ef47b0145c18430a28c0e53a0d6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:25.391047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97896d-d82e-4847-8124-d5018a13ae9a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:25.400563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 664/1324] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:25 2022] DELETE /identity/v3/users/e9740ef47b0145c18430a28c0e53a0d6 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:25.403175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:25.404669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.404968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.405155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743b52cd-5964-4de2-9f30-f9ad84e97363 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.405337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.405443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743b52cd-5964-4de2-9f30-f9ad84e97363 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.405691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743b52cd-5964-4de2-9f30-f9ad84e97363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.405958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:25.408171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:25.446062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 661/1325] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:25 2022] POST /identity/v3/users => generated 308 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:25.451201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:25.458077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:25.468730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-743b52cd-5964-4de2-9f30-f9ad84e97363 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:25.472430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-743b52cd-5964-4de2-9f30-f9ad84e97363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:25.493966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5cb69c-5f9c-4345-802f-1b8c940013eb admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:25.522231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:25.559335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:25.626627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 665/1326] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:25.626627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743b52cd-5964-4de2-9f30-f9ad84e97363 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:25.628901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-743b52cd-5964-4de2-9f30-f9ad84e97363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:25.637147 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:25.638394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.639108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.639807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.641906 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:25.645198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:25.682609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 662/1327] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:25 2022] GET /identity/v3/roles => generated 1420 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:25.684882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:25.689301 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:25.801316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743b52cd-5964-4de2-9f30-f9ad84e97363 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:25.813799 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4747e094d75d4d7b9cc498bf26ea1727', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '61b20b3d0794441ca2f960285a767476', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:25.814596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.814959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.815236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.815990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:25.816318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:25.820853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:25.850049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff01acbf-aef6-4896-9af7-2a5fa3722f3d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:25.853199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 666/1328] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:25.854334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:25.857493 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ca0d08-8cff-4845-9cee-e0662ccd480d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:25.858332 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca0d08-8cff-4845-9cee-e0662ccd480d service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.858735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca0d08-8cff-4845-9cee-e0662ccd480d service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.859103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca0d08-8cff-4845-9cee-e0662ccd480d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.875396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead9a14-d657-4a8b-a728-e99d4ee6295f tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:25.878761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ca0d08-8cff-4845-9cee-e0662ccd480d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:25.881763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ca0d08-8cff-4845-9cee-e0662ccd480d service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:25.891302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76af77f0-c33a-496c-bf74-e24c34cba441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:25.892382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76af77f0-c33a-496c-bf74-e24c34cba441 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.892897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76af77f0-c33a-496c-bf74-e24c34cba441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.893426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76af77f0-c33a-496c-bf74-e24c34cba441 admin admin] PATH_INFO: `/v3/projects/e4d467e038be4d1a9c4598cf327ad535/users/772c6502d8c24d84bb90e22ff7a6f772/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.897959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 663/1329] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:25.898866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ca0d08-8cff-4845-9cee-e0662ccd480d service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:25.922861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76af77f0-c33a-496c-bf74-e24c34cba441 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4d467e038be4d1a9c4598cf327ad535, user_id=772c6502d8c24d84bb90e22ff7a6f772, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:25.923530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ca0d08-8cff-4845-9cee-e0662ccd480d service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:25.926791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76af77f0-c33a-496c-bf74-e24c34cba441 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:25.934262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393b7636-b1fa-4e7a-9d91-a51c0bb336ed tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4747e094d75d4d7b9cc498bf26ea1727', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '61b20b3d0794441ca2f960285a767476', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:25.934922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393b7636-b1fa-4e7a-9d91-a51c0bb336ed tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:25.935212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393b7636-b1fa-4e7a-9d91-a51c0bb336ed tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:25.935492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393b7636-b1fa-4e7a-9d91-a51c0bb336ed tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:25.938541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393b7636-b1fa-4e7a-9d91-a51c0bb336ed tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:25.947930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393b7636-b1fa-4e7a-9d91-a51c0bb336ed tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:25.956014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 667/1330] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:25 2022] PUT /identity/v3/projects/e4d467e038be4d1a9c4598cf327ad535/users/772c6502d8c24d84bb90e22ff7a6f772/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:25.960534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76af77f0-c33a-496c-bf74-e24c34cba441 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:26.009497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 664/1331] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:25 2022] POST /identity/v3/projects => generated 352 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:26.011230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393b7636-b1fa-4e7a-9d91-a51c0bb336ed tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:26.079313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76af77f0-c33a-496c-bf74-e24c34cba441 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:26.131946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393b7636-b1fa-4e7a-9d91-a51c0bb336ed tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:26.139895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4747e094d75d4d7b9cc498bf26ea1727', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '61b20b3d0794441ca2f960285a767476', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:26.141828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:26.142120 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:26.142383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:26.144885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:26.145224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:26.147903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:26.172891 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 665/1332] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:26.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:26.266402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c90a32-90c1-476e-b4ed-495755868697 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:26.269709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8add6e9-6db8-41a9-8744-52290ce4104d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:26.271000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8add6e9-6db8-41a9-8744-52290ce4104d admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:26.272789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8add6e9-6db8-41a9-8744-52290ce4104d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:26.281248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8add6e9-6db8-41a9-8744-52290ce4104d admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:26.282968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8add6e9-6db8-41a9-8744-52290ce4104d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:26.285375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8add6e9-6db8-41a9-8744-52290ce4104d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:26.290161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:26.294511 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4747e094d75d4d7b9cc498bf26ea1727', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '61b20b3d0794441ca2f960285a767476', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:26.304112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:26.306858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:26.311169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 668/1333] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:25 2022] GET /identity/v3/roles => generated 1420 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:40:26.315150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8add6e9-6db8-41a9-8744-52290ce4104d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:26.328988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] PATH_INFO: `/v3/projects/babdbd6af45740439314067d4086d78b` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:26.335851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Authorizing `identity:delete_project(project_id=babdbd6af45740439314067d4086d78b)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:26.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8add6e9-6db8-41a9-8744-52290ce4104d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:26.352206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:26.360556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:26.361355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:26.363266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:26.363666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] PATH_INFO: `/v3/projects/e4d467e038be4d1a9c4598cf327ad535/users/772c6502d8c24d84bb90e22ff7a6f772/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:26.369729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4d467e038be4d1a9c4598cf327ad535, user_id=772c6502d8c24d84bb90e22ff7a6f772, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:26.375876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:26.401708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 669/1334] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:26 2022] PUT /identity/v3/projects/e4d467e038be4d1a9c4598cf327ad535/users/772c6502d8c24d84bb90e22ff7a6f772/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:26.415576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:26.453605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'babdbd6af45740439314067d4086d78b', 'request_id': 'req-7f57ea1c-0025-4721-866e-1f45a37ea353'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:26.470754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'babdbd6af45740439314067d4086d78b', 'request_id': 'req-7f57ea1c-0025-4721-866e-1f45a37ea353'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:26.486476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'babdbd6af45740439314067d4086d78b', 'request_id': 'req-7f57ea1c-0025-4721-866e-1f45a37ea353'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:26.511681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] The token cache is being invalidate because project babdbd6af45740439314067d4086d78b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:26.512101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:26.512842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7f57ea1c-0025-4721-866e-1f45a37ea353 tempest-VolumeQuotaClassesTest-1247324262 tempest-VolumeQuotaClassesTest-1247324262-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:26.521348 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 666/1335] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:26 2022] DELETE /identity/v3/projects/babdbd6af45740439314067d4086d78b => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:40:26.945635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:26.948045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:27.092445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79408a76-73fa-4b78-aa1a-4e2840919f72 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:27.236061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:27.237947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:27.238404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:27.239072 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] PATH_INFO: `/v3/users/ae9312d545d645c3a44d5ae0fbb99fbe` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:27.241310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae9312d545d645c3a44d5ae0fbb99fbe)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:27.244264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:27.345885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae9312d545d645c3a44d5ae0fbb99fbe'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.358581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae9312d545d645c3a44d5ae0fbb99fbe'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.359468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae9312d545d645c3a44d5ae0fbb99fbe'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.360237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae9312d545d645c3a44d5ae0fbb99fbe'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.370207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae9312d545d645c3a44d5ae0fbb99fbe'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.413782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 670/1336] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:26 2022] DELETE /identity/v3/users/ae9312d545d645c3a44d5ae0fbb99fbe => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:27.419000 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26b466-b406-4163-bd77-910523d798ad None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:27.419936 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26b466-b406-4163-bd77-910523d798ad None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:27.420656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26b466-b406-4163-bd77-910523d798ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:27.451086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6a26b466-b406-4163-bd77-910523d798ad None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:27.452800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6a26b466-b406-4163-bd77-910523d798ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:27.477304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:27.479656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:27.527541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:27.668634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 667/1337] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:27.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a26b466-b406-4163-bd77-910523d798ad None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:27.683881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6a26b466-b406-4163-bd77-910523d798ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:27.705948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185d3fc0-53c0-48f2-a4ed-6adb83378a89 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:27.713535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:27.714555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:27.715407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:27.716163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] PATH_INFO: `/v3/projects/9e9333ca01214933af63450c15a2b51b` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:27.721280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e9333ca01214933af63450c15a2b51b)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:27.723965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: WARNING py.warnings [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:40:27.723965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: warnings.warn(msg) Oct 19 00:40:27.723965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: Oct 19 00:40:27.727135 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:27.790244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a26b466-b406-4163-bd77-910523d798ad None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:27.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e527a359-d8d0-481e-a2e9-ff719f550b97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:27.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e527a359-d8d0-481e-a2e9-ff719f550b97 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:27.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e527a359-d8d0-481e-a2e9-ff719f550b97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:27.809400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e527a359-d8d0-481e-a2e9-ff719f550b97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:27.841015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e527a359-d8d0-481e-a2e9-ff719f550b97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:27.842810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e527a359-d8d0-481e-a2e9-ff719f550b97 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:27.855626 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e9333ca01214933af63450c15a2b51b', 'request_id': 'req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.869581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 668/1338] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:27.878839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e9333ca01214933af63450c15a2b51b', 'request_id': 'req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.898335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e9333ca01214933af63450c15a2b51b', 'request_id': 'req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.948990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] The token cache is being invalidate because project 9e9333ca01214933af63450c15a2b51b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:27.950533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.951586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a57e4722-e5d1-4fce-b697-b5228d7e7f0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:27.959091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 671/1339] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:27 2022] DELETE /identity/v3/projects/9e9333ca01214933af63450c15a2b51b => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:27.960684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33abe3a9-2319-41f4-abfc-df6a51f08dea None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:27.960975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33abe3a9-2319-41f4-abfc-df6a51f08dea None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:27.961265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33abe3a9-2319-41f4-abfc-df6a51f08dea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:28.032565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-33abe3a9-2319-41f4-abfc-df6a51f08dea None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:28.037767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-33abe3a9-2319-41f4-abfc-df6a51f08dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:28.158979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:28.184819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 669/1340] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:40:28.190202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac97d18-a87d-47bf-b207-01ddc03daa8b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:28.190882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac97d18-a87d-47bf-b207-01ddc03daa8b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:28.191333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac97d18-a87d-47bf-b207-01ddc03daa8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:28.231519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-fac97d18-a87d-47bf-b207-01ddc03daa8b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:28.234137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fac97d18-a87d-47bf-b207-01ddc03daa8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:28.251846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33abe3a9-2319-41f4-abfc-df6a51f08dea None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:28.253487 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 672/1341] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:28.265487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33abe3a9-2319-41f4-abfc-df6a51f08dea None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:28.271916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:28.272921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:28.273094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:28.273275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:28.274085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:28.274392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:28.276313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:28.295742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 670/1342] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:28.297828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac97d18-a87d-47bf-b207-01ddc03daa8b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:28.312710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac97d18-a87d-47bf-b207-01ddc03daa8b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:28.335420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79fbeea-07c3-40bf-841d-5f73cc216c2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:28.337100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79fbeea-07c3-40bf-841d-5f73cc216c2e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:28.338589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79fbeea-07c3-40bf-841d-5f73cc216c2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:28.339397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79fbeea-07c3-40bf-841d-5f73cc216c2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:28.341618 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79fbeea-07c3-40bf-841d-5f73cc216c2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:28.344881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79fbeea-07c3-40bf-841d-5f73cc216c2e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:28.484096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:28.487457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 673/1343] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:28 2022] POST /identity/v3/projects => generated 365 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:28.497619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eca3d5e-cc3a-4f33-b5ce-d62adb49b3de admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:28.644469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e490e3-f897-4f93-8c63-ea20882ce6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:28.645266 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e490e3-f897-4f93-8c63-ea20882ce6ed admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:28.645559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e490e3-f897-4f93-8c63-ea20882ce6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:28.645902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e490e3-f897-4f93-8c63-ea20882ce6ed admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:28.646581 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e490e3-f897-4f93-8c63-ea20882ce6ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:28.649829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e490e3-f897-4f93-8c63-ea20882ce6ed admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:28.709712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 671/1344] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:28 2022] POST /identity/v3/users => generated 283 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:28.714659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79fbeea-07c3-40bf-841d-5f73cc216c2e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:28.733704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79fbeea-07c3-40bf-841d-5f73cc216c2e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:28.756133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666083b-ed03-45fb-be32-4d06c9bf5ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:28.760352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666083b-ed03-45fb-be32-4d06c9bf5ef9 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:28.761813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666083b-ed03-45fb-be32-4d06c9bf5ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:28.763184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666083b-ed03-45fb-be32-4d06c9bf5ef9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:28.766715 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666083b-ed03-45fb-be32-4d06c9bf5ef9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:28.771716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666083b-ed03-45fb-be32-4d06c9bf5ef9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:28.792760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 674/1345] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:28 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:28.797107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e490e3-f897-4f93-8c63-ea20882ce6ed admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:28.813536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e490e3-f897-4f93-8c63-ea20882ce6ed admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:28.821013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f985d9-4e3c-4776-9769-9936095f64fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:28.822171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f985d9-4e3c-4776-9769-9936095f64fb admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:28.822659 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f985d9-4e3c-4776-9769-9936095f64fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:28.823205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f985d9-4e3c-4776-9769-9936095f64fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:28.824093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f985d9-4e3c-4776-9769-9936095f64fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:28.825992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f985d9-4e3c-4776-9769-9936095f64fb admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:28.855060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666083b-ed03-45fb-be32-4d06c9bf5ef9 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:28.868588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 672/1346] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:28 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:28.870375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666083b-ed03-45fb-be32-4d06c9bf5ef9 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:28.882528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122c7c33-25c3-4d43-88f9-70b950f94a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:28.884200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122c7c33-25c3-4d43-88f9-70b950f94a41 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:28.884854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122c7c33-25c3-4d43-88f9-70b950f94a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:28.885303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122c7c33-25c3-4d43-88f9-70b950f94a41 admin admin] PATH_INFO: `/v3/projects/282ecec4c2304d62975cc0be219461ad/users/64d48e7156664ad1b6d98380791292dc/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:28.919558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122c7c33-25c3-4d43-88f9-70b950f94a41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=282ecec4c2304d62975cc0be219461ad, user_id=64d48e7156664ad1b6d98380791292dc, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:28.924153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122c7c33-25c3-4d43-88f9-70b950f94a41 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:28.966970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 675/1347] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:28 2022] PUT /identity/v3/projects/282ecec4c2304d62975cc0be219461ad/users/64d48e7156664ad1b6d98380791292dc/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:28.969042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7b1b1e-8496-4b5b-9c51-d2c249cf84c7 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:28.969596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7b1b1e-8496-4b5b-9c51-d2c249cf84c7 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:28.970037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7b1b1e-8496-4b5b-9c51-d2c249cf84c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:29.021246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6a7b1b1e-8496-4b5b-9c51-d2c249cf84c7 None None] MFA Rules not processed for user `64d48e7156664ad1b6d98380791292dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:29.024154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6a7b1b1e-8496-4b5b-9c51-d2c249cf84c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:29.159713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 673/1348] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:28 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:29.162050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122c7c33-25c3-4d43-88f9-70b950f94a41 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:29.250196 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:29.306302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122c7c33-25c3-4d43-88f9-70b950f94a41 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:29.321714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288209c4-5e49-4bea-9efa-52957e28b2e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:29.322771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288209c4-5e49-4bea-9efa-52957e28b2e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:29.323412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288209c4-5e49-4bea-9efa-52957e28b2e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:29.323849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288209c4-5e49-4bea-9efa-52957e28b2e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:29.325074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288209c4-5e49-4bea-9efa-52957e28b2e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:29.327543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288209c4-5e49-4bea-9efa-52957e28b2e1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:29.379249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7b1b1e-8496-4b5b-9c51-d2c249cf84c7 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:29.381868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6a7b1b1e-8496-4b5b-9c51-d2c249cf84c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:29.421202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 676/1349] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:29 2022] POST /identity/v3/projects => generated 365 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:40:29.430751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288209c4-5e49-4bea-9efa-52957e28b2e1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:29.441328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7b1b1e-8496-4b5b-9c51-d2c249cf84c7 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:29.640603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:29.655094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288209c4-5e49-4bea-9efa-52957e28b2e1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:29.666202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:29.667252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:29.667674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:29.667956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] PATH_INFO: `/v3/projects/8ed39b7029ea4681aaa9939701c317c1` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:29.672577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ed39b7029ea4681aaa9939701c317c1)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:29.674008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e2aed9-cbdd-47b1-a814-2d69b8237ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:29.675439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2aed9-cbdd-47b1-a814-2d69b8237ffd admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:29.675908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2aed9-cbdd-47b1-a814-2d69b8237ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:29.676262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2aed9-cbdd-47b1-a814-2d69b8237ffd admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:29.679111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e2aed9-cbdd-47b1-a814-2d69b8237ffd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:29.684392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e2aed9-cbdd-47b1-a814-2d69b8237ffd admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:29.684594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:29.742032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 677/1350] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:29 2022] POST /identity/v3/users => generated 289 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:29.743781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd8748-8ec2-4ee3-86f2-b553394f4fbf None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:29.745609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd8748-8ec2-4ee3-86f2-b553394f4fbf None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:29.746019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd8748-8ec2-4ee3-86f2-b553394f4fbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:29.857374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-6dfd8748-8ec2-4ee3-86f2-b553394f4fbf None None] MFA Rules not processed for user `772c6502d8c24d84bb90e22ff7a6f772`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:29.859078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6dfd8748-8ec2-4ee3-86f2-b553394f4fbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:29.904459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ed39b7029ea4681aaa9939701c317c1', 'request_id': 'req-460a994c-c620-416d-baba-4cbb7280f262'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:29.935692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ed39b7029ea4681aaa9939701c317c1', 'request_id': 'req-460a994c-c620-416d-baba-4cbb7280f262'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:29.951054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ed39b7029ea4681aaa9939701c317c1', 'request_id': 'req-460a994c-c620-416d-baba-4cbb7280f262'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:29.976295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] The token cache is being invalidate because project 8ed39b7029ea4681aaa9939701c317c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:29.976607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:29.977355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:29.983062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 674/1351] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:29 2022] DELETE /identity/v3/projects/8ed39b7029ea4681aaa9939701c317c1 => generated 0 bytes in 604 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:29.983991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:29.991415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:30.076074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 678/1352] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:29 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:30.077324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfd8748-8ec2-4ee3-86f2-b553394f4fbf None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:30.081724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6dfd8748-8ec2-4ee3-86f2-b553394f4fbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:30.151783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a994c-c620-416d-baba-4cbb7280f262 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:30.181174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099d2381-6ca0-4d83-b037-c8591d105c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:30.181819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099d2381-6ca0-4d83-b037-c8591d105c0b admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:30.182103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099d2381-6ca0-4d83-b037-c8591d105c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:30.182391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099d2381-6ca0-4d83-b037-c8591d105c0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:30.183347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099d2381-6ca0-4d83-b037-c8591d105c0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:30.185844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099d2381-6ca0-4d83-b037-c8591d105c0b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:30.212363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 675/1353] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:29 2022] GET /identity/v3/roles => generated 1420 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:30.255750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:30.265244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099d2381-6ca0-4d83-b037-c8591d105c0b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:30.294881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099d2381-6ca0-4d83-b037-c8591d105c0b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:30.310649 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156519d4-a75c-457d-831f-88564ac96297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:30.311736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156519d4-a75c-457d-831f-88564ac96297 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:30.312184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156519d4-a75c-457d-831f-88564ac96297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:30.312653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156519d4-a75c-457d-831f-88564ac96297 admin admin] PATH_INFO: `/v3/projects/4579a0bebd0e41549c86f118e30e227d/users/90b98f99121343ea8e923f3dfd5488d6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:30.339587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156519d4-a75c-457d-831f-88564ac96297 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4579a0bebd0e41549c86f118e30e227d, user_id=90b98f99121343ea8e923f3dfd5488d6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:30.345952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156519d4-a75c-457d-831f-88564ac96297 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:30.383912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 676/1354] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:30 2022] PUT /identity/v3/projects/4579a0bebd0e41549c86f118e30e227d/users/90b98f99121343ea8e923f3dfd5488d6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:30.384878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfd8748-8ec2-4ee3-86f2-b553394f4fbf None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:30.397230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156519d4-a75c-457d-831f-88564ac96297 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:30.561929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:30.574895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156519d4-a75c-457d-831f-88564ac96297 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:30.598616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:30.602439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:30.602793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:30.603362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] PATH_INFO: `/v3/users/4747e094d75d4d7b9cc498bf26ea1727` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:30.605461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4747e094d75d4d7b9cc498bf26ea1727)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:30.610693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbc55a1-c43c-4c36-b4d4-697ef9b800ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:30.612827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:30.613295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbc55a1-c43c-4c36-b4d4-697ef9b800ea admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:30.614037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbc55a1-c43c-4c36-b4d4-697ef9b800ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:30.614594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbc55a1-c43c-4c36-b4d4-697ef9b800ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:30.617793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbc55a1-c43c-4c36-b4d4-697ef9b800ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:30.646974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbc55a1-c43c-4c36-b4d4-697ef9b800ea admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:30.699533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 677/1355] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:30 2022] GET /identity/v3/roles => generated 1420 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:40:30.701014 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbc55a1-c43c-4c36-b4d4-697ef9b800ea admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:30.735031 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbc55a1-c43c-4c36-b4d4-697ef9b800ea admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:30.746139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f8991-e5b0-465e-8ef0-02639dabbd96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:30.747241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f8991-e5b0-465e-8ef0-02639dabbd96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:30.747756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f8991-e5b0-465e-8ef0-02639dabbd96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:30.748275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f8991-e5b0-465e-8ef0-02639dabbd96 admin admin] PATH_INFO: `/v3/projects/4579a0bebd0e41549c86f118e30e227d/users/90b98f99121343ea8e923f3dfd5488d6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:30.753402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5f8991-e5b0-465e-8ef0-02639dabbd96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4579a0bebd0e41549c86f118e30e227d, user_id=90b98f99121343ea8e923f3dfd5488d6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:30.757479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4747e094d75d4d7b9cc498bf26ea1727'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:30.775799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5f8991-e5b0-465e-8ef0-02639dabbd96 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:30.780897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4747e094d75d4d7b9cc498bf26ea1727'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:30.781816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4747e094d75d4d7b9cc498bf26ea1727'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:30.782624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4747e094d75d4d7b9cc498bf26ea1727'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:30.800442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4747e094d75d4d7b9cc498bf26ea1727'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:30.834065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 678/1356] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:30 2022] PUT /identity/v3/projects/4579a0bebd0e41549c86f118e30e227d/users/90b98f99121343ea8e923f3dfd5488d6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:30.835796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1207f5-0273-4f9b-8446-ddbe03803f35 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:30.836239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1207f5-0273-4f9b-8446-ddbe03803f35 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:30.836545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1207f5-0273-4f9b-8446-ddbe03803f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:30.864351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 679/1357] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:30 2022] DELETE /identity/v3/users/4747e094d75d4d7b9cc498bf26ea1727 => generated 0 bytes in 787 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:30.893515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5d1207f5-0273-4f9b-8446-ddbe03803f35 None None] MFA Rules not processed for user `90b98f99121343ea8e923f3dfd5488d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:30.895926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5d1207f5-0273-4f9b-8446-ddbe03803f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:31.091559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 679/1358] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:30 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:40:31.092756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:31.094907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:31.242632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320d6dbd-ac4a-4be5-a400-f0f26994c194 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:31.248596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:31.249254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:31.249538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:31.249813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:31.292360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:31.293588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:31.325340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 680/1359] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:31 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:40:31.534895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1207f5-0273-4f9b-8446-ddbe03803f35 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:31.555063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1207f5-0273-4f9b-8446-ddbe03803f35 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:31.564828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b19ad-6bb0-4cba-af4d-d4e3e47da43e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:31.565494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b19ad-6bb0-4cba-af4d-d4e3e47da43e service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:31.566129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b19ad-6bb0-4cba-af4d-d4e3e47da43e service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:31.566412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b19ad-6bb0-4cba-af4d-d4e3e47da43e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:31.572182 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-368b19ad-6bb0-4cba-af4d-d4e3e47da43e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:31.718081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b19ad-6bb0-4cba-af4d-d4e3e47da43e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:31.719933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b19ad-6bb0-4cba-af4d-d4e3e47da43e service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:31.749634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 680/1360] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:31 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:32.704981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:32.707560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:32.734242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:32.871375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bd9ea5-4022-48ab-9a8e-2d17b36a47f9 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:32.886409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:32.887408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:32.887690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:32.887963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] PATH_INFO: `/v3/projects/61b20b3d0794441ca2f960285a767476` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:32.889178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=61b20b3d0794441ca2f960285a767476)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:32.891281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:32.994024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61b20b3d0794441ca2f960285a767476', 'request_id': 'req-f410d754-b18c-47ae-ab78-afb5972ce1ce'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:33.002359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61b20b3d0794441ca2f960285a767476', 'request_id': 'req-f410d754-b18c-47ae-ab78-afb5972ce1ce'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:33.019129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61b20b3d0794441ca2f960285a767476', 'request_id': 'req-f410d754-b18c-47ae-ab78-afb5972ce1ce'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:33.038166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] The token cache is being invalidate because project 61b20b3d0794441ca2f960285a767476 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:33.038804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:33.039694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f410d754-b18c-47ae-ab78-afb5972ce1ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:33.041762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 681/1361] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:32 2022] DELETE /identity/v3/projects/61b20b3d0794441ca2f960285a767476 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:33.051771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b8821-7b65-4934-853d-fb8cff740180 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.051771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b8821-7b65-4934-853d-fb8cff740180 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.051771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b8821-7b65-4934-853d-fb8cff740180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.114314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-028b8821-7b65-4934-853d-fb8cff740180 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:33.115894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-028b8821-7b65-4934-853d-fb8cff740180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:33.248269 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 681/1362] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:33.257288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aa654b-f9bf-4e8d-93c4-d35fe389c4a4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.257941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aa654b-f9bf-4e8d-93c4-d35fe389c4a4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.258702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aa654b-f9bf-4e8d-93c4-d35fe389c4a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.308242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f2aa654b-f9bf-4e8d-93c4-d35fe389c4a4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:33.310025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f2aa654b-f9bf-4e8d-93c4-d35fe389c4a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:33.333200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 682/1363] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:33.337351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b8821-7b65-4934-853d-fb8cff740180 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:33.343299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:33.361572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b8821-7b65-4934-853d-fb8cff740180 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:33.369545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:33.370461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.371138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.371589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.372403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:33.372859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:33.375321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:33.390642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 682/1364] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:33.396144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aa654b-f9bf-4e8d-93c4-d35fe389c4a4 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:33.416324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aa654b-f9bf-4e8d-93c4-d35fe389c4a4 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:33.423639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e991cc-0edb-4b33-8886-22de05d2635f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:33.424577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e991cc-0edb-4b33-8886-22de05d2635f admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.425451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e991cc-0edb-4b33-8886-22de05d2635f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.425940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e991cc-0edb-4b33-8886-22de05d2635f admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.427045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e991cc-0edb-4b33-8886-22de05d2635f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:33.430789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e991cc-0edb-4b33-8886-22de05d2635f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:33.480027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 683/1365] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:33 2022] POST /identity/v3/projects => generated 383 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:33.483950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:33.499419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4379c858-8e8f-40c5-adf9-fa926b8322ee admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:33.617171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52ccded-2f29-45c6-af30-1d864f1bb44a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:33.621026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52ccded-2f29-45c6-af30-1d864f1bb44a admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.622871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52ccded-2f29-45c6-af30-1d864f1bb44a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.624525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52ccded-2f29-45c6-af30-1d864f1bb44a admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.627203 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52ccded-2f29-45c6-af30-1d864f1bb44a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:33.634078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52ccded-2f29-45c6-af30-1d864f1bb44a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:33.696515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 683/1366] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:33 2022] POST /identity/v3/users => generated 292 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:33.698645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e991cc-0edb-4b33-8886-22de05d2635f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:33.719111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e991cc-0edb-4b33-8886-22de05d2635f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:33.733205 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb6347-d973-4762-9ad7-dbf7564af5fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:33.733911 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb6347-d973-4762-9ad7-dbf7564af5fa admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.734199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb6347-d973-4762-9ad7-dbf7564af5fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.735317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb6347-d973-4762-9ad7-dbf7564af5fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.736029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb6347-d973-4762-9ad7-dbf7564af5fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:33.743651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb6347-d973-4762-9ad7-dbf7564af5fa admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:33.769577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 684/1367] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:33 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:33.771699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52ccded-2f29-45c6-af30-1d864f1bb44a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:33.792513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52ccded-2f29-45c6-af30-1d864f1bb44a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:33.807477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d734e69-5dba-44cc-b803-68b35f443778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:33.810060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d734e69-5dba-44cc-b803-68b35f443778 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.810411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d734e69-5dba-44cc-b803-68b35f443778 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.810729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d734e69-5dba-44cc-b803-68b35f443778 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.812260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d734e69-5dba-44cc-b803-68b35f443778 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:33.817004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d734e69-5dba-44cc-b803-68b35f443778 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:33.851639 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 684/1368] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:33 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:40:33.853387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb6347-d973-4762-9ad7-dbf7564af5fa admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:33.873533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb6347-d973-4762-9ad7-dbf7564af5fa admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:33.896294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7d39da-556f-42d4-9f8d-fd1c75a855c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:33.896993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d39da-556f-42d4-9f8d-fd1c75a855c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.897275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d39da-556f-42d4-9f8d-fd1c75a855c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.897622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d39da-556f-42d4-9f8d-fd1c75a855c9 admin admin] PATH_INFO: `/v3/projects/17d5d65b675649f68539f4f4abb1897e/users/2c83f784f7174253a64e4a3eee9edba3/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.933036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a6eef0-c0b9-4bb4-b03b-fdce7a328e0b None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.933848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a6eef0-c0b9-4bb4-b03b-fdce7a328e0b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.934128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a6eef0-c0b9-4bb4-b03b-fdce7a328e0b None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:33.936210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 685/1369] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:40:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:40:33.949759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7d39da-556f-42d4-9f8d-fd1c75a855c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17d5d65b675649f68539f4f4abb1897e, user_id=2c83f784f7174253a64e4a3eee9edba3, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:33.953450 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7d39da-556f-42d4-9f8d-fd1c75a855c9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:33.988075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 685/1370] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:33 2022] PUT /identity/v3/projects/17d5d65b675649f68539f4f4abb1897e/users/2c83f784f7174253a64e4a3eee9edba3/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:40:33.997999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0427ac-2a5a-4a44-bc45-6f3586c93825 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:33.998308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0427ac-2a5a-4a44-bc45-6f3586c93825 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:33.998576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0427ac-2a5a-4a44-bc45-6f3586c93825 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:34.077489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7e0427ac-2a5a-4a44-bc45-6f3586c93825 None None] MFA Rules not processed for user `2c83f784f7174253a64e4a3eee9edba3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:34.084734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7e0427ac-2a5a-4a44-bc45-6f3586c93825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:34.266151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85caa1f8-b18c-4e93-9471-a0d51401d7de None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:34.266675 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85caa1f8-b18c-4e93-9471-a0d51401d7de None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:34.267268 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85caa1f8-b18c-4e93-9471-a0d51401d7de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:34.336658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 686/1371] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:33 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:34.338297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0427ac-2a5a-4a44-bc45-6f3586c93825 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:34.342914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-85caa1f8-b18c-4e93-9471-a0d51401d7de None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:34.351909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-85caa1f8-b18c-4e93-9471-a0d51401d7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:34.504841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:34.612791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:34.627317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0427ac-2a5a-4a44-bc45-6f3586c93825 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:34.648569 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:34.649308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:34.649662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:34.650010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:34.650788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:34.652827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:34.663704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 686/1372] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:40:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 399 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:40:34.735997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 687/1373] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:34 2022] POST /identity/v3/projects => generated 381 bytes in 398 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:34.741166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85caa1f8-b18c-4e93-9471-a0d51401d7de None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:34.760094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85caa1f8-b18c-4e93-9471-a0d51401d7de None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:34.824240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:34.824536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:34.907824 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af5b9a8-22ed-4e9c-b723-2d0a76e068d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:34.908579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5b9a8-22ed-4e9c-b723-2d0a76e068d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:34.909651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5b9a8-22ed-4e9c-b723-2d0a76e068d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:34.910141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5b9a8-22ed-4e9c-b723-2d0a76e068d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:34.910946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af5b9a8-22ed-4e9c-b723-2d0a76e068d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:34.913013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af5b9a8-22ed-4e9c-b723-2d0a76e068d1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:34.969951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 687/1374] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:34 2022] POST /identity/v3/users => generated 297 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:40:34.974772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af5b9a8-22ed-4e9c-b723-2d0a76e068d1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:34.993054 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af5b9a8-22ed-4e9c-b723-2d0a76e068d1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:35.004859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886ccc3e-9bf7-404e-87aa-6abee36c3bc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:35.005630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886ccc3e-9bf7-404e-87aa-6abee36c3bc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:35.005959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac1f96-48b6-49fb-8e79-702a6d78f5b1 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:35.006104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886ccc3e-9bf7-404e-87aa-6abee36c3bc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:35.006408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886ccc3e-9bf7-404e-87aa-6abee36c3bc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:35.007162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886ccc3e-9bf7-404e-87aa-6abee36c3bc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:35.008873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886ccc3e-9bf7-404e-87aa-6abee36c3bc6 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:35.017520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b1514-e644-4df9-8d13-ca0096de962d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:35.018708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b1514-e644-4df9-8d13-ca0096de962d service placement] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:35.018992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b1514-e644-4df9-8d13-ca0096de962d service placement] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:35.019267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b1514-e644-4df9-8d13-ca0096de962d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:35.036891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 688/1375] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:34 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:40:35.040869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886ccc3e-9bf7-404e-87aa-6abee36c3bc6 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:35.046249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0b1514-e644-4df9-8d13-ca0096de962d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:35.049543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0b1514-e644-4df9-8d13-ca0096de962d service placement] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:35.056429 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886ccc3e-9bf7-404e-87aa-6abee36c3bc6 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:35.074785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9e60d-2270-4501-9a34-8014d35ecd64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:35.075481 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9e60d-2270-4501-9a34-8014d35ecd64 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:35.075876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9e60d-2270-4501-9a34-8014d35ecd64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:35.076229 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9e60d-2270-4501-9a34-8014d35ecd64 admin admin] PATH_INFO: `/v3/projects/5de2f93c9def4c2393a4a775ca9cfb70/users/13713997e6634679b2cf193afa55132d/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:35.086913 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 688/1376] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:40:35.110826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb9e60d-2270-4501-9a34-8014d35ecd64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5de2f93c9def4c2393a4a775ca9cfb70, user_id=13713997e6634679b2cf193afa55132d, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:35.114038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb9e60d-2270-4501-9a34-8014d35ecd64 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:35.138113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 689/1377] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:35 2022] PUT /identity/v3/projects/5de2f93c9def4c2393a4a775ca9cfb70/users/13713997e6634679b2cf193afa55132d/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:40:35.141975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b1514-e644-4df9-8d13-ca0096de962d service placement] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:35.173535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b1514-e644-4df9-8d13-ca0096de962d service placement] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:35.309959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794d29e6-1d14-4745-8805-e2664aea6219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:35.311100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794d29e6-1d14-4745-8805-e2664aea6219 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:35.312768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794d29e6-1d14-4745-8805-e2664aea6219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:35.313061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794d29e6-1d14-4745-8805-e2664aea6219 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:35.314730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794d29e6-1d14-4745-8805-e2664aea6219 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:35.322904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794d29e6-1d14-4745-8805-e2664aea6219 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:35.366714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 689/1378] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:35 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:35.368492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9e60d-2270-4501-9a34-8014d35ecd64 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:35.387146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9e60d-2270-4501-9a34-8014d35ecd64 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:35.399168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:35.399797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:35.400079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:35.400356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] PATH_INFO: `/v3/projects/5de2f93c9def4c2393a4a775ca9cfb70/users/13713997e6634679b2cf193afa55132d/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:35.402436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5de2f93c9def4c2393a4a775ca9cfb70, user_id=13713997e6634679b2cf193afa55132d, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:35.405180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:35.444564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 690/1379] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:35 2022] PUT /identity/v3/projects/5de2f93c9def4c2393a4a775ca9cfb70/users/13713997e6634679b2cf193afa55132d/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:35.455578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047bc3da-f7da-46c5-be6f-2f910ba3b425 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:35.456773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047bc3da-f7da-46c5-be6f-2f910ba3b425 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:35.457058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047bc3da-f7da-46c5-be6f-2f910ba3b425 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:35.557970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-047bc3da-f7da-46c5-be6f-2f910ba3b425 None None] MFA Rules not processed for user `13713997e6634679b2cf193afa55132d`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:35.559758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-047bc3da-f7da-46c5-be6f-2f910ba3b425 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:35.732473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:35.757628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 690/1380] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:35 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 303 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:35.765658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:35.767633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:35.953006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c6b7e7-4752-4ef2-a6f3-45e5b511af3a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:35.964134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4411f5bb-07d6-4463-b040-9cc49ede8a23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:35.964733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411f5bb-07d6-4463-b040-9cc49ede8a23 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:35.965015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411f5bb-07d6-4463-b040-9cc49ede8a23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:35.965290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411f5bb-07d6-4463-b040-9cc49ede8a23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:35.985419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4411f5bb-07d6-4463-b040-9cc49ede8a23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:35.986914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4411f5bb-07d6-4463-b040-9cc49ede8a23 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:36.026649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 691/1381] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:35 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:36.249637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047bc3da-f7da-46c5-be6f-2f910ba3b425 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:36.251814 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-047bc3da-f7da-46c5-be6f-2f910ba3b425 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:36.413633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047bc3da-f7da-46c5-be6f-2f910ba3b425 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:36.425162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4411f5bb-07d6-4463-b040-9cc49ede8a23 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:36.425612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:36.426246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:36.426564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:36.426846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] PATH_INFO: `/v3/users/70a99ac47e8447668deee708fadcbb7d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:36.431316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=70a99ac47e8447668deee708fadcbb7d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:36.432445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: WARNING py.warnings [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 00:40:36.432445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: warnings.warn(msg) Oct 19 00:40:36.432445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: Oct 19 00:40:36.433767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:36.436607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4411f5bb-07d6-4463-b040-9cc49ede8a23 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:36.448250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:36.449423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:36.449719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:36.450926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:36.469788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:36.588729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:36.590201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:36.605559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70a99ac47e8447668deee708fadcbb7d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.637478 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 692/1382] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:36.637478 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:36.645081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:36.652356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70a99ac47e8447668deee708fadcbb7d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.655978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70a99ac47e8447668deee708fadcbb7d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.658325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70a99ac47e8447668deee708fadcbb7d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.692307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6c589318-2831-4d27-9a84-e784c567e1fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70a99ac47e8447668deee708fadcbb7d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.701694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5313bec4-fa54-4bc2-99f4-125f70a3d96c service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:36.723748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:36.726903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:36.727664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:36.727952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] PATH_INFO: `/v3/users/64d48e7156664ad1b6d98380791292dc` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:36.733735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] RBAC: Authorizing `identity:delete_user(user_id=64d48e7156664ad1b6d98380791292dc)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:36.743131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:36.798193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 691/1383] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:36 2022] DELETE /identity/v3/users/70a99ac47e8447668deee708fadcbb7d => generated 0 bytes in 549 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:36.887548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '64d48e7156664ad1b6d98380791292dc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.904117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '64d48e7156664ad1b6d98380791292dc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.907305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '64d48e7156664ad1b6d98380791292dc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.908571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '64d48e7156664ad1b6d98380791292dc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.925123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '64d48e7156664ad1b6d98380791292dc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:36.975833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 693/1384] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:36 2022] DELETE /identity/v3/users/64d48e7156664ad1b6d98380791292dc => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:36.980699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ae6d2c-2600-4ffd-b9b5-8f67f060340b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:36.981004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ae6d2c-2600-4ffd-b9b5-8f67f060340b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:36.981250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ae6d2c-2600-4ffd-b9b5-8f67f060340b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:37.031535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-61ae6d2c-2600-4ffd-b9b5-8f67f060340b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:37.040485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-61ae6d2c-2600-4ffd-b9b5-8f67f060340b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:37.246413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 692/1385] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:37.441431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:37.442951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:37.597037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1dd83-a413-4fa2-9d44-96121aa3c30f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:37.610214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d8442e-4708-4a07-b629-e263920e5cd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:37.610711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d8442e-4708-4a07-b629-e263920e5cd0 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:37.610993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d8442e-4708-4a07-b629-e263920e5cd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:37.611949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d8442e-4708-4a07-b629-e263920e5cd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:37.645144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d8442e-4708-4a07-b629-e263920e5cd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:37.646811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d8442e-4708-4a07-b629-e263920e5cd0 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:37.668638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 694/1386] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:37.749976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ae6d2c-2600-4ffd-b9b5-8f67f060340b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:37.752566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-61ae6d2c-2600-4ffd-b9b5-8f67f060340b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:37.767980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ae6d2c-2600-4ffd-b9b5-8f67f060340b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:37.778868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:37.779908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:37.780310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:37.780757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] PATH_INFO: `/v3/users/2b8ffb23ca6a446493020dfd68fc9eae` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:37.782356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b8ffb23ca6a446493020dfd68fc9eae)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:37.785588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:37.793978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:37.906656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b8ffb23ca6a446493020dfd68fc9eae'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:37.918957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b8ffb23ca6a446493020dfd68fc9eae'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:37.920070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b8ffb23ca6a446493020dfd68fc9eae'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:37.920942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b8ffb23ca6a446493020dfd68fc9eae'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:37.951498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-defe7a6f-1cf1-40cf-a3bb-2ffce6b29a18 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b8ffb23ca6a446493020dfd68fc9eae'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:38.014861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184b99a-8a44-4f92-ae88-b5b218dff11d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:38.015492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184b99a-8a44-4f92-ae88-b5b218dff11d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:38.015492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184b99a-8a44-4f92-ae88-b5b218dff11d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:38.016407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 693/1387] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:37 2022] DELETE /identity/v3/users/2b8ffb23ca6a446493020dfd68fc9eae => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:38.058326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-d184b99a-8a44-4f92-ae88-b5b218dff11d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:38.060376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d184b99a-8a44-4f92-ae88-b5b218dff11d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:38.107039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:38.208295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954a162-20ec-447e-80d1-e75b4f395244 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:38.209140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954a162-20ec-447e-80d1-e75b4f395244 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:38.209578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954a162-20ec-447e-80d1-e75b4f395244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:38.224651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 695/1388] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:40:38.242730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d184b99a-8a44-4f92-ae88-b5b218dff11d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:38.244543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d184b99a-8a44-4f92-ae88-b5b218dff11d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:38.328801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d954a162-20ec-447e-80d1-e75b4f395244 None None] MFA Rules not processed for user `d3f0d615187d4779bdd7da8c9f24c796`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:38.333465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d954a162-20ec-447e-80d1-e75b4f395244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:38.404503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d184b99a-8a44-4f92-ae88-b5b218dff11d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:38.422371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:38.437842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:38.438442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:38.440920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:38.511182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:38.513277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:38.556104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 696/1389] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:40:38.560753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:38.564441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:38.603864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 694/1390] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:38 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 400 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:38.605812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838336a3-5303-4572-a160-d052148c3795 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:38.605812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838336a3-5303-4572-a160-d052148c3795 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:38.606067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838336a3-5303-4572-a160-d052148c3795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:38.621091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8218053-4d68-40f4-8287-30c2a87cfa76 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:38.651789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:38.652615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:38.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:38.653390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] PATH_INFO: `/v3/projects/47e9d7d5512148d5ae8eb9dda5b3657d` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:38.660124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47e9d7d5512148d5ae8eb9dda5b3657d)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:38.663635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:38.695580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-838336a3-5303-4572-a160-d052148c3795 None None] MFA Rules not processed for user `d3f0d615187d4779bdd7da8c9f24c796`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:38.697233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-838336a3-5303-4572-a160-d052148c3795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:38.720175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 695/1391] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:38 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:38.729258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838336a3-5303-4572-a160-d052148c3795 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:38.730853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-838336a3-5303-4572-a160-d052148c3795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:38.866187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47e9d7d5512148d5ae8eb9dda5b3657d', 'request_id': 'req-0947f91e-9b32-4660-becc-ef0d7a633d94'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:38.879551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47e9d7d5512148d5ae8eb9dda5b3657d', 'request_id': 'req-0947f91e-9b32-4660-becc-ef0d7a633d94'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:38.891407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47e9d7d5512148d5ae8eb9dda5b3657d', 'request_id': 'req-0947f91e-9b32-4660-becc-ef0d7a633d94'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:38.926270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] The token cache is being invalidate because project 47e9d7d5512148d5ae8eb9dda5b3657d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:38.926676 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:38.931891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:38.936445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 697/1392] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:38 2022] DELETE /identity/v3/projects/47e9d7d5512148d5ae8eb9dda5b3657d => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:38.938030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:38.941537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:38.957228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838336a3-5303-4572-a160-d052148c3795 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:38.967172 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:39.068504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:39.101492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0947f91e-9b32-4660-becc-ef0d7a633d94 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:39.105186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0da10f-f643-49c4-bc9a-8a8836a6ebf5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:39.105813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0da10f-f643-49c4-bc9a-8a8836a6ebf5 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:39.105964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0da10f-f643-49c4-bc9a-8a8836a6ebf5 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:39.106222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0da10f-f643-49c4-bc9a-8a8836a6ebf5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:39.109192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:39.110162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:39.111383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7e0da10f-f643-49c4-bc9a-8a8836a6ebf5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:39.112617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:39.113490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] PATH_INFO: `/v3/users/772c6502d8c24d84bb90e22ff7a6f772` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:39.116079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] RBAC: Authorizing `identity:delete_user(user_id=772c6502d8c24d84bb90e22ff7a6f772)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:39.118934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:39.248529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '772c6502d8c24d84bb90e22ff7a6f772'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:39.248785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0da10f-f643-49c4-bc9a-8a8836a6ebf5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:39.253158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0da10f-f643-49c4-bc9a-8a8836a6ebf5 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:39.264644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '772c6502d8c24d84bb90e22ff7a6f772'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:39.265420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '772c6502d8c24d84bb90e22ff7a6f772'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:39.266087 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '772c6502d8c24d84bb90e22ff7a6f772'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:39.268963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 696/1393] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:38 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 540 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:39.277665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '772c6502d8c24d84bb90e22ff7a6f772'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:39.316702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 698/1394] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:38 2022] DELETE /identity/v3/users/772c6502d8c24d84bb90e22ff7a6f772 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:40:39.531310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765e89ee-c638-4252-a33a-9bdd33251b38 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:39.531667 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765e89ee-c638-4252-a33a-9bdd33251b38 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:39.532222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765e89ee-c638-4252-a33a-9bdd33251b38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:39.614820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-765e89ee-c638-4252-a33a-9bdd33251b38 None None] MFA Rules not processed for user `69fef5650fff42e4ac4ed3ca74d4b123`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:39.616298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-765e89ee-c638-4252-a33a-9bdd33251b38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:39.767023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 697/1395] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:39 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:39.775232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:39.776954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:39.841615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765e89ee-c638-4252-a33a-9bdd33251b38 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:39.844644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-765e89ee-c638-4252-a33a-9bdd33251b38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:39.956499 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:40.070056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139ab5c2-968b-4909-9c91-430a08a55828 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:40.090646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:40.094540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:40.097920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:40.098422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:40.111970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765e89ee-c638-4252-a33a-9bdd33251b38 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:40.121548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:40.124996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:40.126184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:40.127107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:40.127464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:40.127744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] PATH_INFO: `/v3/projects/282ecec4c2304d62975cc0be219461ad` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:40.136411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=282ecec4c2304d62975cc0be219461ad)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:40.138379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:40.155590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 699/1396] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:39 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:40.156783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:40.161934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:40.204385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60858864-c785-4fb2-a954-8a3f9ea26223 service nova] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:40.220084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:40.222750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:40.227139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:40.227412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] PATH_INFO: `/v3/projects/a9ebac69f755473cae560ff06fa55e00` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:40.228932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9ebac69f755473cae560ff06fa55e00)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:40.234950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:40.239042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '282ecec4c2304d62975cc0be219461ad', 'request_id': 'req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.251481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '282ecec4c2304d62975cc0be219461ad', 'request_id': 'req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.267178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '282ecec4c2304d62975cc0be219461ad', 'request_id': 'req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.289336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] The token cache is being invalidate because project 282ecec4c2304d62975cc0be219461ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:40.289635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.291304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.294582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 698/1397] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:39 2022] DELETE /identity/v3/projects/282ecec4c2304d62975cc0be219461ad => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:40.295559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:40.298530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:40.386624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9ebac69f755473cae560ff06fa55e00', 'request_id': 'req-73760089-a738-47c2-81bc-78d84ec43aed'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.391658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:40.406147 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9ebac69f755473cae560ff06fa55e00', 'request_id': 'req-73760089-a738-47c2-81bc-78d84ec43aed'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.441225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9ebac69f755473cae560ff06fa55e00', 'request_id': 'req-73760089-a738-47c2-81bc-78d84ec43aed'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.449103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee05b7d-9bd0-4571-8ff0-c5020ec147f4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:40.472113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] The token cache is being invalidate because project a9ebac69f755473cae560ff06fa55e00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:40.472692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.475053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.477782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 700/1398] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:40 2022] DELETE /identity/v3/projects/a9ebac69f755473cae560ff06fa55e00 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:40.479545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:40.483394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:40.617296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:40.618100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:40.618387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:40.618818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] PATH_INFO: `/v3/projects/e4d467e038be4d1a9c4598cf327ad535` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:40.621291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4d467e038be4d1a9c4598cf327ad535)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:40.627032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:40.657431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:40.689654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73760089-a738-47c2-81bc-78d84ec43aed admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:40.696168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:40.696833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:40.697673 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:40.697971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] PATH_INFO: `/v3/users/2c83f784f7174253a64e4a3eee9edba3` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:40.700045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c83f784f7174253a64e4a3eee9edba3)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:40.715135 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:40.747219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4d467e038be4d1a9c4598cf327ad535', 'request_id': 'req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.765074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4d467e038be4d1a9c4598cf327ad535', 'request_id': 'req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.793364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4d467e038be4d1a9c4598cf327ad535', 'request_id': 'req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.807865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c83f784f7174253a64e4a3eee9edba3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.812141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] The token cache is being invalidate because project e4d467e038be4d1a9c4598cf327ad535 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:40.812643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.813551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.815854 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 699/1399] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:40 2022] DELETE /identity/v3/projects/e4d467e038be4d1a9c4598cf327ad535 => generated 0 bytes in 520 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:40.816700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:40.818587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:40.843561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c83f784f7174253a64e4a3eee9edba3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.844523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c83f784f7174253a64e4a3eee9edba3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.845168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c83f784f7174253a64e4a3eee9edba3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.856316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c83f784f7174253a64e4a3eee9edba3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:40.896094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 701/1400] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:40 2022] DELETE /identity/v3/users/2c83f784f7174253a64e4a3eee9edba3 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:40.897090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:40.908500 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:40.973219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a41ef-7ff8-4583-b0d7-0f58f69eaa0f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:41.090626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:41.094024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:41.094508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:41.094798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] PATH_INFO: `/v3/users/90b98f99121343ea8e923f3dfd5488d6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:41.097066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] RBAC: Authorizing `identity:delete_user(user_id=90b98f99121343ea8e923f3dfd5488d6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:41.099737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:41.140519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b0e79a-dead-4212-b96b-37d5ffcdfc9e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:41.150729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:41.151400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:41.151680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:41.152006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] PATH_INFO: `/v3/users/ffbead23e0104ab2a2f4a9373b6c1ebc` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:41.154861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffbead23e0104ab2a2f4a9373b6c1ebc)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:41.157048 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:41.158939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:41.199333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90b98f99121343ea8e923f3dfd5488d6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.207519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:41.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90b98f99121343ea8e923f3dfd5488d6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.237531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90b98f99121343ea8e923f3dfd5488d6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.244490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90b98f99121343ea8e923f3dfd5488d6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.266584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b949c30-b277-4dab-b619-cab7b692c02a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90b98f99121343ea8e923f3dfd5488d6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.276567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffbead23e0104ab2a2f4a9373b6c1ebc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.313623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 700/1401] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:40 2022] DELETE /identity/v3/users/90b98f99121343ea8e923f3dfd5488d6 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:41.315445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d856a2-f479-4012-b1e9-a737065df519 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:41.315916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d856a2-f479-4012-b1e9-a737065df519 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:41.316209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d856a2-f479-4012-b1e9-a737065df519 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:41.320839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffbead23e0104ab2a2f4a9373b6c1ebc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.321807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffbead23e0104ab2a2f4a9373b6c1ebc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.321914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 701/1402] 10.222.0.26 () {54 vars in 912 bytes} [Wed Oct 19 00:40:41 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:40:41.323325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e9ec49-b589-4bea-95ad-33cd7c9100b4 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:41.323620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e9ec49-b589-4bea-95ad-33cd7c9100b4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:41.323799 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffbead23e0104ab2a2f4a9373b6c1ebc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.323898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e9ec49-b589-4bea-95ad-33cd7c9100b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:41.347425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4e421645-d9f4-4ba7-a896-d625660ba75b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffbead23e0104ab2a2f4a9373b6c1ebc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:41.405926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 702/1403] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:40 2022] DELETE /identity/v3/users/ffbead23e0104ab2a2f4a9373b6c1ebc => generated 0 bytes in 507 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:41.405926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4fc832-4a33-4634-8b83-63d00c93b73a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:41.405926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4fc832-4a33-4634-8b83-63d00c93b73a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:41.420479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4fc832-4a33-4634-8b83-63d00c93b73a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:41.434934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e7e9ec49-b589-4bea-95ad-33cd7c9100b4 None None] MFA Rules not processed for user `6f8725c4d05b4dd79b05b43fba15980b`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:41.436205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e7e9ec49-b589-4bea-95ad-33cd7c9100b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:41.501144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8d4fc832-4a33-4634-8b83-63d00c93b73a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:41.519843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8d4fc832-4a33-4634-8b83-63d00c93b73a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:41.701413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 702/1404] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:41 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 378 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:41.703428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0fe7f3-2d00-4ffa-85ef-befa38f8b3cd None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:41.706330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0fe7f3-2d00-4ffa-85ef-befa38f8b3cd None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:41.706903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0fe7f3-2d00-4ffa-85ef-befa38f8b3cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:41.710079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 703/1405] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:41.712055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e589c8-b0e7-488e-9dc1-9f8c7d01caf7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:41.714925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e589c8-b0e7-488e-9dc1-9f8c7d01caf7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:41.715465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e589c8-b0e7-488e-9dc1-9f8c7d01caf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:41.823770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-cf0fe7f3-2d00-4ffa-85ef-befa38f8b3cd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:41.826238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-cf0fe7f3-2d00-4ffa-85ef-befa38f8b3cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:41.853737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 703/1406] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:41.857979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ec085f-eca8-4f0c-8e86-9414f5aa0f33 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:41.857979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ec085f-eca8-4f0c-8e86-9414f5aa0f33 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:41.859496 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ec085f-eca8-4f0c-8e86-9414f5aa0f33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:41.911011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-86ec085f-eca8-4f0c-8e86-9414f5aa0f33 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:41.912256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-86ec085f-eca8-4f0c-8e86-9414f5aa0f33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:41.927029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a9e589c8-b0e7-488e-9dc1-9f8c7d01caf7 None None] MFA Rules not processed for user `5756f4ff604a413b8af1f12c82cc4870`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:41.931056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a9e589c8-b0e7-488e-9dc1-9f8c7d01caf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:41.932743 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 704/1407] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:41.934300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ec085f-eca8-4f0c-8e86-9414f5aa0f33 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:41.938462 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-86ec085f-eca8-4f0c-8e86-9414f5aa0f33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:42.055791 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 704/1408] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:40:41 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:42.068288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfee5da-ed7d-47dc-a036-dc87de83b28c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.068605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfee5da-ed7d-47dc-a036-dc87de83b28c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.068780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfee5da-ed7d-47dc-a036-dc87de83b28c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.179461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-dbfee5da-ed7d-47dc-a036-dc87de83b28c None None] MFA Rules not processed for user `8118f0c698aa428a93cb054abac30c2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:42.182296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-dbfee5da-ed7d-47dc-a036-dc87de83b28c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:42.188333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:42.203115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ec085f-eca8-4f0c-8e86-9414f5aa0f33 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:42.253002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:42.258528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.258885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.259167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.291430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:42.293075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:42.344479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 705/1409] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:42 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:42.355078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 705/1410] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:41 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:42.355078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:42.356827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:42.371595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfee5da-ed7d-47dc-a036-dc87de83b28c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:42.399247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb893e9-b2cc-488e-85d6-20d93a16513c service nova] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:42.410647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfee5da-ed7d-47dc-a036-dc87de83b28c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:42.410998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f64e90-36b0-4b78-9ab8-1c554016c44c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:42.411625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f64e90-36b0-4b78-9ab8-1c554016c44c service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.411903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f64e90-36b0-4b78-9ab8-1c554016c44c service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.412536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f64e90-36b0-4b78-9ab8-1c554016c44c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.429737 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a5c175-c552-4573-9391-3aa00093c52a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:42.430830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a5c175-c552-4573-9391-3aa00093c52a admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.430830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a5c175-c552-4573-9391-3aa00093c52a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.431021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a5c175-c552-4573-9391-3aa00093c52a admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.431638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a5c175-c552-4573-9391-3aa00093c52a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:42.431942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a5c175-c552-4573-9391-3aa00093c52a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:42.433605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a5c175-c552-4573-9391-3aa00093c52a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:42.463331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 706/1411] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:42.464843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ae376-c1c8-4460-8a53-f8fb2f39c845 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.465136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ae376-c1c8-4460-8a53-f8fb2f39c845 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.465410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ae376-c1c8-4460-8a53-f8fb2f39c845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.498665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f64e90-36b0-4b78-9ab8-1c554016c44c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:42.502966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f64e90-36b0-4b78-9ab8-1c554016c44c service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:42.539357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 706/1412] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:42.540331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f64e90-36b0-4b78-9ab8-1c554016c44c service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:42.542216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c34ae376-c1c8-4460-8a53-f8fb2f39c845 None None] MFA Rules not processed for user `8118f0c698aa428a93cb054abac30c2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:42.543795 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c34ae376-c1c8-4460-8a53-f8fb2f39c845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:42.562330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f64e90-36b0-4b78-9ab8-1c554016c44c service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:42.577821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 707/1413] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:42 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:42.579003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34ae376-c1c8-4460-8a53-f8fb2f39c845 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:42.593524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34ae376-c1c8-4460-8a53-f8fb2f39c845 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:42.605048 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7919916-5cac-41a2-bc3f-482f13a4775a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:42.605267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46524e8-cf45-4619-93e4-11e0190ccb6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:42.605723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46524e8-cf45-4619-93e4-11e0190ccb6d admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.606005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46524e8-cf45-4619-93e4-11e0190ccb6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.606284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46524e8-cf45-4619-93e4-11e0190ccb6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.607123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46524e8-cf45-4619-93e4-11e0190ccb6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:42.607404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7919916-5cac-41a2-bc3f-482f13a4775a service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.607885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7919916-5cac-41a2-bc3f-482f13a4775a service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.609112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46524e8-cf45-4619-93e4-11e0190ccb6d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:42.609604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7919916-5cac-41a2-bc3f-482f13a4775a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.629433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7919916-5cac-41a2-bc3f-482f13a4775a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:42.632293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7919916-5cac-41a2-bc3f-482f13a4775a service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:42.663651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 708/1414] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:42.664594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7919916-5cac-41a2-bc3f-482f13a4775a service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:42.672040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:42.702398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 707/1415] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:42 2022] POST /identity/v3/projects => generated 383 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:42.707181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46524e8-cf45-4619-93e4-11e0190ccb6d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:42.707864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7919916-5cac-41a2-bc3f-482f13a4775a service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:42.862410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d92561-7239-4286-83c2-abda73efd677 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:42.863391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d92561-7239-4286-83c2-abda73efd677 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.873377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d92561-7239-4286-83c2-abda73efd677 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.873701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d92561-7239-4286-83c2-abda73efd677 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.884597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46524e8-cf45-4619-93e4-11e0190ccb6d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:42.899583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37286614-38a8-4c97-b1f0-589667d16aac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:42.900233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37286614-38a8-4c97-b1f0-589667d16aac admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.900577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37286614-38a8-4c97-b1f0-589667d16aac admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.900896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37286614-38a8-4c97-b1f0-589667d16aac admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.901557 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37286614-38a8-4c97-b1f0-589667d16aac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:42.903599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37286614-38a8-4c97-b1f0-589667d16aac admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:42.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 708/1416] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:42 2022] POST /identity/v3/users => generated 292 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:42.958048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37286614-38a8-4c97-b1f0-589667d16aac admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:42.983123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37286614-38a8-4c97-b1f0-589667d16aac admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:42.994930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f12eaf6-7b2e-4e04-a8c5-f92bf75e1a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:42.997324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f12eaf6-7b2e-4e04-a8c5-f92bf75e1a6f admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:42.997324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f12eaf6-7b2e-4e04-a8c5-f92bf75e1a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:42.997529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f12eaf6-7b2e-4e04-a8c5-f92bf75e1a6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:42.998381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f12eaf6-7b2e-4e04-a8c5-f92bf75e1a6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:43.001919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f12eaf6-7b2e-4e04-a8c5-f92bf75e1a6f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:43.022157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d92561-7239-4286-83c2-abda73efd677 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:43.023906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d92561-7239-4286-83c2-abda73efd677 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:43.036483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 709/1417] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:43.045077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f12eaf6-7b2e-4e04-a8c5-f92bf75e1a6f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:43.047184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 709/1418] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:42 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:43.084426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f12eaf6-7b2e-4e04-a8c5-f92bf75e1a6f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:43.121308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:43.122180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:43.122180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:43.122764 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:43.123473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:43.127499 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:43.155853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 710/1419] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:43.159409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d92561-7239-4286-83c2-abda73efd677 service nova] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:43.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d92561-7239-4286-83c2-abda73efd677 service nova] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:43.196447 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:43.197137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:43.197424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:43.197754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] PATH_INFO: `/v3/projects/326db2eede964aec9dbd06693dd47cb1/users/ce0717e10bb24cbca0b23361e62c1359/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:43.202439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:43.229041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=326db2eede964aec9dbd06693dd47cb1, user_id=ce0717e10bb24cbca0b23361e62c1359, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:43.235570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:43.235752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:43.238759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:43.265749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97d1514-54df-4cc0-a893-41c133364135 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:43.296247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 710/1420] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:43 2022] PUT /identity/v3/projects/326db2eede964aec9dbd06693dd47cb1/users/ce0717e10bb24cbca0b23361e62c1359/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:43.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:43.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:43.426446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:43.428677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:43.430940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:43.433318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] PATH_INFO: `/v3/projects/4579a0bebd0e41549c86f118e30e227d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:43.437890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4579a0bebd0e41549c86f118e30e227d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:43.444517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:43.534553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85369392-c861-4792-8dad-63b4092491d2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:43.558961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:43.560190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:43.560938 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:43.561547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] PATH_INFO: `/v3/projects/28a5bc93e44c492e8c8c13426423b02a` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:43.567272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=28a5bc93e44c492e8c8c13426423b02a)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:43.570358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:43.625580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4579a0bebd0e41549c86f118e30e227d', 'request_id': 'req-d4f86ab2-1335-477e-916f-7fee9199ae79'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.641358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4579a0bebd0e41549c86f118e30e227d', 'request_id': 'req-d4f86ab2-1335-477e-916f-7fee9199ae79'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.651624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28a5bc93e44c492e8c8c13426423b02a', 'request_id': 'req-74331610-6903-4fe4-bab1-b1a53e243f6b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.658310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4579a0bebd0e41549c86f118e30e227d', 'request_id': 'req-d4f86ab2-1335-477e-916f-7fee9199ae79'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.661426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:INSERT=4,keystone:SELECT=90,keystone:DELETE=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:43.661742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28a5bc93e44c492e8c8c13426423b02a', 'request_id': 'req-74331610-6903-4fe4-bab1-b1a53e243f6b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.675103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28a5bc93e44c492e8c8c13426423b02a', 'request_id': 'req-74331610-6903-4fe4-bab1-b1a53e243f6b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.708576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] The token cache is being invalidate because project 28a5bc93e44c492e8c8c13426423b02a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:43.708887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.709495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] The token cache is being invalidate because project 4579a0bebd0e41549c86f118e30e227d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:43.709608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-74331610-6903-4fe4-bab1-b1a53e243f6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.709800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.712423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d4f86ab2-1335-477e-916f-7fee9199ae79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:43.714537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 711/1421] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:43 2022] DELETE /identity/v3/projects/28a5bc93e44c492e8c8c13426423b02a => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:43.723788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913127e2-caef-43c6-ab31-f06ad16eea42 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:43.724090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913127e2-caef-43c6-ab31-f06ad16eea42 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:43.724376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913127e2-caef-43c6-ab31-f06ad16eea42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:43.728652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 711/1422] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:43 2022] DELETE /identity/v3/projects/4579a0bebd0e41549c86f118e30e227d => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:43.728652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e546a8-f4b9-442f-ac32-9f0f2c67a7ee None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:43.728652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e546a8-f4b9-442f-ac32-9f0f2c67a7ee None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:43.728652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e546a8-f4b9-442f-ac32-9f0f2c67a7ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:43.797787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-913127e2-caef-43c6-ab31-f06ad16eea42 None None] MFA Rules not processed for user `ce0717e10bb24cbca0b23361e62c1359`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:43.800803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-913127e2-caef-43c6-ab31-f06ad16eea42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:43.825305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-34e546a8-f4b9-442f-ac32-9f0f2c67a7ee None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:43.828810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-34e546a8-f4b9-442f-ac32-9f0f2c67a7ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:43.969446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 712/1423] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:43.972409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f8847-baf8-4b0e-912b-7c061c184a1d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:43.973038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f8847-baf8-4b0e-912b-7c061c184a1d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:43.975476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f8847-baf8-4b0e-912b-7c061c184a1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.020571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-1e3f8847-baf8-4b0e-912b-7c061c184a1d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:44.022708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 712/1424] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:44.024610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd36b4-8831-43d5-aac0-d577fcd9e900 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.024791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1e3f8847-baf8-4b0e-912b-7c061c184a1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:44.024901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd36b4-8831-43d5-aac0-d577fcd9e900 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.025203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd36b4-8831-43d5-aac0-d577fcd9e900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.067887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 713/1425] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:44.069099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3f8847-baf8-4b0e-912b-7c061c184a1d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.077900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1e3f8847-baf8-4b0e-912b-7c061c184a1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:44.079712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:44.091064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b2cd36b4-8831-43d5-aac0-d577fcd9e900 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:44.094698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b2cd36b4-8831-43d5-aac0-d577fcd9e900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:44.108128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3f8847-baf8-4b0e-912b-7c061c184a1d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:44.117433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8d5384-08e1-4e94-a1fa-d231d2285400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:44.117803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 713/1426] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:44.118432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d5384-08e1-4e94-a1fa-d231d2285400 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.119017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d5384-08e1-4e94-a1fa-d231d2285400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.119603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff741e-8e5e-4387-a25d-8b8f3f469c09 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.119966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff741e-8e5e-4387-a25d-8b8f3f469c09 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.120325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff741e-8e5e-4387-a25d-8b8f3f469c09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.120879 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8d5384-08e1-4e94-a1fa-d231d2285400 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.121684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8d5384-08e1-4e94-a1fa-d231d2285400 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:44.125563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8d5384-08e1-4e94-a1fa-d231d2285400 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:44.146461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9fff741e-8e5e-4387-a25d-8b8f3f469c09 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:44.147932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9fff741e-8e5e-4387-a25d-8b8f3f469c09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:44.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 714/1427] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:44.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fff741e-8e5e-4387-a25d-8b8f3f469c09 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.163517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9fff741e-8e5e-4387-a25d-8b8f3f469c09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:44.183464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 714/1428] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:44 2022] POST /identity/v3/projects => generated 383 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:44.191661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8d5384-08e1-4e94-a1fa-d231d2285400 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.290098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fff741e-8e5e-4387-a25d-8b8f3f469c09 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:44.300708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8d5384-08e1-4e94-a1fa-d231d2285400 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:44.317977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:44.318824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.319223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.319605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.325417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:44.327494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:44.329262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:44.344176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 715/1429] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:44.349115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.361017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1862c1ff-de88-45ef-8c1f-0f27c4844c02 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:44.373265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:44.374080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.374458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.375209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.375440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:44.375761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:44.380086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:44.408968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 716/1430] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:44.408968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.429452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660efdc4-49e4-47d7-9fcd-1851c7f6beba admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:44.439586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf05ee6c-1f25-44a8-a22f-5e13c07fe9d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:44.440464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf05ee6c-1f25-44a8-a22f-5e13c07fe9d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.440638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:44.440755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf05ee6c-1f25-44a8-a22f-5e13c07fe9d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.440955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf05ee6c-1f25-44a8-a22f-5e13c07fe9d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.441602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf05ee6c-1f25-44a8-a22f-5e13c07fe9d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:44.441705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.442158 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.442549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.443660 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf05ee6c-1f25-44a8-a22f-5e13c07fe9d3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:44.449174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:44.467116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:44.470317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:44.475308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 717/1431] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:44 2022] POST /identity/v3/users => generated 298 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:44.476683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a9181b-2ce9-4964-b64b-cf1d67b059dc None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.476980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a9181b-2ce9-4964-b64b-cf1d67b059dc None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.477326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a9181b-2ce9-4964-b64b-cf1d67b059dc None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.478976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 718/1432] 10.222.0.26 () {54 vars in 912 bytes} [Wed Oct 19 00:40:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:40:44.480819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec42a0c0-14a5-430d-988c-7d2228b166b6 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.481149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec42a0c0-14a5-430d-988c-7d2228b166b6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.481433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec42a0c0-14a5-430d-988c-7d2228b166b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.495104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 715/1433] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:44.495964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.510760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c561aeee-e80d-4d8d-98ca-2f85e4079952 service placement] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:44.511428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ec42a0c0-14a5-430d-988c-7d2228b166b6 None None] MFA Rules not processed for user `d3f0d615187d4779bdd7da8c9f24c796`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:44.512661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ec42a0c0-14a5-430d-988c-7d2228b166b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:44.516152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d5de8f-38c6-4610-abdf-8beab03d7e88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:44.517164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d5de8f-38c6-4610-abdf-8beab03d7e88 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.518343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d5de8f-38c6-4610-abdf-8beab03d7e88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.518343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d5de8f-38c6-4610-abdf-8beab03d7e88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.521374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d5de8f-38c6-4610-abdf-8beab03d7e88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:44.527680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d5de8f-38c6-4610-abdf-8beab03d7e88 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:44.548762 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:INSERT=5,keystone:DELETE=1,keystone:SELECT=94 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:44.601803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 716/1434] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:44 2022] POST /identity/v3/projects => generated 357 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:44.601803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d5de8f-38c6-4610-abdf-8beab03d7e88 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.633017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 719/1435] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:44 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:44.634311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec42a0c0-14a5-430d-988c-7d2228b166b6 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.651426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec42a0c0-14a5-430d-988c-7d2228b166b6 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:44.837689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d5de8f-38c6-4610-abdf-8beab03d7e88 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:44.855621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:44.859532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.862350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.863014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.864748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:44.872683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:44.881741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f3206c-cfdd-48cc-9d34-9b3f565e9b9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:44.882590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f3206c-cfdd-48cc-9d34-9b3f565e9b9c admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.884536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f3206c-cfdd-48cc-9d34-9b3f565e9b9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.884947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f3206c-cfdd-48cc-9d34-9b3f565e9b9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.890049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f3206c-cfdd-48cc-9d34-9b3f565e9b9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:44.892352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f3206c-cfdd-48cc-9d34-9b3f565e9b9c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:44.893718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:44.934722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 720/1436] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:44.937175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ced7180-dc32-44f5-976d-84087695c461 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:44.938357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ced7180-dc32-44f5-976d-84087695c461 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:44.940807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ced7180-dc32-44f5-976d-84087695c461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:44.966352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 717/1437] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:44 2022] POST /identity/v3/projects => generated 367 bytes in 365 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:44.969008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:44.976558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.004375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5ced7180-dc32-44f5-976d-84087695c461 None None] MFA Rules not processed for user `5756f4ff604a413b8af1f12c82cc4870`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:45.011440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5ced7180-dc32-44f5-976d-84087695c461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.146746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 721/1438] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:40:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:45.147771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ced7180-dc32-44f5-976d-84087695c461 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:45.163961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ced7180-dc32-44f5-976d-84087695c461 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.170617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1ada28-2867-4088-9470-e778c5a01856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.171586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1ada28-2867-4088-9470-e778c5a01856 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.172165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1ada28-2867-4088-9470-e778c5a01856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.172698 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1ada28-2867-4088-9470-e778c5a01856 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.173510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1ada28-2867-4088-9470-e778c5a01856 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.176262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1ada28-2867-4088-9470-e778c5a01856 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.182985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7621bd1-495d-4b45-bcb3-c4b6c3bf70e0 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.193331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.195190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.195508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.195835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] PATH_INFO: `/v3/projects/17d5d65b675649f68539f4f4abb1897e` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.204769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17d5d65b675649f68539f4f4abb1897e)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.209075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.209221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 722/1439] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:45 2022] POST /identity/v3/users => generated 279 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:45.210417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1ada28-2867-4088-9470-e778c5a01856 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:45.227385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1ada28-2867-4088-9470-e778c5a01856 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.234587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe560db-0b6b-4d6d-9b90-381f05d63528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.235533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe560db-0b6b-4d6d-9b90-381f05d63528 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.236066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe560db-0b6b-4d6d-9b90-381f05d63528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.236446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe560db-0b6b-4d6d-9b90-381f05d63528 admin admin] PATH_INFO: `/v3/projects/7317ead796c04b7593525ce05e02505c/users/3804dc9cbb1c4661b58840115d305942/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.311529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe560db-0b6b-4d6d-9b90-381f05d63528 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7317ead796c04b7593525ce05e02505c, user_id=3804dc9cbb1c4661b58840115d305942, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.316042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe560db-0b6b-4d6d-9b90-381f05d63528 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.343953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 723/1440] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:45 2022] PUT /identity/v3/projects/7317ead796c04b7593525ce05e02505c/users/3804dc9cbb1c4661b58840115d305942/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:45.345387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe560db-0b6b-4d6d-9b90-381f05d63528 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:45.428871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:45.429912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17d5d65b675649f68539f4f4abb1897e', 'request_id': 'req-b60d15e0-061d-43be-9690-bafb620eb007'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:45.446474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17d5d65b675649f68539f4f4abb1897e', 'request_id': 'req-b60d15e0-061d-43be-9690-bafb620eb007'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:45.463554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17d5d65b675649f68539f4f4abb1897e', 'request_id': 'req-b60d15e0-061d-43be-9690-bafb620eb007'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:45.469579 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe560db-0b6b-4d6d-9b90-381f05d63528 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.491761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] The token cache is being invalidate because project 17d5d65b675649f68539f4f4abb1897e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:45.492075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:45.497598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:45.500937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 718/1441] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:44 2022] DELETE /identity/v3/projects/17d5d65b675649f68539f4f4abb1897e => generated 0 bytes in 534 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:45.501779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:45.506136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.577274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.578226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.578804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.579253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.580007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.583397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.639760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 724/1442] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:45 2022] POST /identity/v3/users => generated 284 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:45.641293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:45.643972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.657324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbaf5cf-d49a-48ac-bb72-a54d85a2f44f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.664628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2befab-21af-4b40-9408-e958689a435b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.665687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2befab-21af-4b40-9408-e958689a435b admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.666118 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2befab-21af-4b40-9408-e958689a435b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.666620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2befab-21af-4b40-9408-e958689a435b admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.667418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2befab-21af-4b40-9408-e958689a435b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.669186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2befab-21af-4b40-9408-e958689a435b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.694123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60d15e0-061d-43be-9690-bafb620eb007 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.695761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 725/1443] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:45.698928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff19e69-b8e0-43d1-afab-f53d5939ebac None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.699397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff19e69-b8e0-43d1-afab-f53d5939ebac None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.699818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff19e69-b8e0-43d1-afab-f53d5939ebac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.704296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.705144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.706108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.707653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.711834 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.739863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.742240 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.746876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7ff19e69-b8e0-43d1-afab-f53d5939ebac None None] MFA Rules not processed for user `69fef5650fff42e4ac4ed3ca74d4b123`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:45.748155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7ff19e69-b8e0-43d1-afab-f53d5939ebac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.765898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 719/1444] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:45.770164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:45.772658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.807018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742054ee-84fe-40e0-94fd-af901bb41119 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.823393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.824166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.824295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.824571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.825299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.827362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.851843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 720/1445] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:45.851843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:45.858512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.871240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 726/1446] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:45 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:45.871240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff19e69-b8e0-43d1-afab-f53d5939ebac None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:45.880543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7ff19e69-b8e0-43d1-afab-f53d5939ebac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:45.887122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8f34b8-7049-4688-8711-79bebbce5a6e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.905401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.906083 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.906388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.906933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] PATH_INFO: `/v3/users/13713997e6634679b2cf193afa55132d` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.908917 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=13713997e6634679b2cf193afa55132d)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.911401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.916164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:45.929971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff19e69-b8e0-43d1-afab-f53d5939ebac None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:45.943539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03da3b1d-48c7-47ea-b11e-732d233d0a65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:45.946295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03da3b1d-48c7-47ea-b11e-732d233d0a65 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:45.946610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03da3b1d-48c7-47ea-b11e-732d233d0a65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:45.947058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03da3b1d-48c7-47ea-b11e-732d233d0a65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:45.949014 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03da3b1d-48c7-47ea-b11e-732d233d0a65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:45.954381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03da3b1d-48c7-47ea-b11e-732d233d0a65 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:45.990074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 727/1447] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:46.000426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03da3b1d-48c7-47ea-b11e-732d233d0a65 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:46.030854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13713997e6634679b2cf193afa55132d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:46.031466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03da3b1d-48c7-47ea-b11e-732d233d0a65 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:46.047768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:46.048904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13713997e6634679b2cf193afa55132d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:46.050585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13713997e6634679b2cf193afa55132d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:46.050839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:46.051121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:46.051397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:46.052057 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:46.060332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13713997e6634679b2cf193afa55132d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:46.079740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:46.081697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13713997e6634679b2cf193afa55132d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:46.113444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 728/1448] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:46.119076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:46.125664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:46.152448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 721/1449] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:45 2022] DELETE /identity/v3/users/13713997e6634679b2cf193afa55132d => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:46.152448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:46.163215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:46.186740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ef753f-d20e-46ec-943a-ab31dba98a3d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:46.253336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:46.294915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:46.295388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:46.297789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:46.298108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] PATH_INFO: `/v3/projects/7317ead796c04b7593525ce05e02505c/users/3804dc9cbb1c4661b58840115d305942/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:46.308193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7317ead796c04b7593525ce05e02505c, user_id=3804dc9cbb1c4661b58840115d305942, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:46.324235 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:46.359544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 729/1450] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:46 2022] PUT /identity/v3/projects/7317ead796c04b7593525ce05e02505c/users/3804dc9cbb1c4661b58840115d305942/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:46.362063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:46.368672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:46.372979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40f5e4-3d45-4608-82d5-3339efee5d7c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:46.618261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96255ca-178b-4494-a7b3-04fabf4db826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:46.619080 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96255ca-178b-4494-a7b3-04fabf4db826 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:46.619176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96255ca-178b-4494-a7b3-04fabf4db826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:46.619452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96255ca-178b-4494-a7b3-04fabf4db826 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:46.622129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96255ca-178b-4494-a7b3-04fabf4db826 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:46.624683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96255ca-178b-4494-a7b3-04fabf4db826 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:46.646608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 722/1451] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 496 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:46.649541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af17efc-0ab5-440c-bc05-12c4ed09c88f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:46.649833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af17efc-0ab5-440c-bc05-12c4ed09c88f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:46.650143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af17efc-0ab5-440c-bc05-12c4ed09c88f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:46.690215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8af17efc-0ab5-440c-bc05-12c4ed09c88f None None] MFA Rules not processed for user `3804dc9cbb1c4661b58840115d305942`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:46.691542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8af17efc-0ab5-440c-bc05-12c4ed09c88f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:46.714241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:46.719575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27d7f43-18b3-40a4-bca2-5a5caee10690 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:46.742075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66557238-6edd-45d4-b156-ad9bf324dd1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:46.742800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66557238-6edd-45d4-b156-ad9bf324dd1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:46.743084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66557238-6edd-45d4-b156-ad9bf324dd1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:46.743363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66557238-6edd-45d4-b156-ad9bf324dd1d admin admin] PATH_INFO: `/v3/projects/9519005aee164bb0a3d0ddda970e4e81/users/9fa5d80ead684ce1a03bd5bdaf30920c/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:46.799735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66557238-6edd-45d4-b156-ad9bf324dd1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9519005aee164bb0a3d0ddda970e4e81, user_id=9fa5d80ead684ce1a03bd5bdaf30920c, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:46.802519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66557238-6edd-45d4-b156-ad9bf324dd1d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:46.847110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 730/1452] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:46 2022] PUT /identity/v3/projects/9519005aee164bb0a3d0ddda970e4e81/users/9fa5d80ead684ce1a03bd5bdaf30920c/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:46.848143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66557238-6edd-45d4-b156-ad9bf324dd1d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:46.939084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 723/1453] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:46 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:46.941318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5b645-3815-4ee9-8c59-027da8f6a387 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:46.941604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5b645-3815-4ee9-8c59-027da8f6a387 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:46.941881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5b645-3815-4ee9-8c59-027da8f6a387 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:47.135288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66557238-6edd-45d4-b156-ad9bf324dd1d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:47.141623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a7e5b645-3815-4ee9-8c59-027da8f6a387 None None] MFA Rules not processed for user `9fa5d80ead684ce1a03bd5bdaf30920c`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:47.146906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a7e5b645-3815-4ee9-8c59-027da8f6a387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:47.150223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:47.151134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:47.151591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:47.152074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] PATH_INFO: `/v3/projects/5e9336eec0d3469ebb7ba20cb4fe8f3c/users/13146d4ec6fa40a2ad1c8ee0647199bf/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:47.185542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e9336eec0d3469ebb7ba20cb4fe8f3c, user_id=13146d4ec6fa40a2ad1c8ee0647199bf, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:47.192546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:47.216842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 731/1454] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:46 2022] PUT /identity/v3/projects/5e9336eec0d3469ebb7ba20cb4fe8f3c/users/13146d4ec6fa40a2ad1c8ee0647199bf/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:47.219976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:47.229919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:47.289308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 724/1455] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:46 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:47.292040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe4b2ec-7dfe-4d66-b40c-af3957e4c25b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:47.292378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe4b2ec-7dfe-4d66-b40c-af3957e4c25b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:47.292774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe4b2ec-7dfe-4d66-b40c-af3957e4c25b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:47.340318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:47.381543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea1aa97-1613-4097-90d8-a842ccadf8f8 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:47.393492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:47.393817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8fe4b2ec-7dfe-4d66-b40c-af3957e4c25b None None] MFA Rules not processed for user `13146d4ec6fa40a2ad1c8ee0647199bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:47.393971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:47.393971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:47.394166 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:47.399017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8fe4b2ec-7dfe-4d66-b40c-af3957e4c25b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:47.594000 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:47.599847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:47.649992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 725/1456] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:47 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 360 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:47.651932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 732/1457] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:47 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:47.652908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe4b2ec-7dfe-4d66-b40c-af3957e4c25b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:47.653233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:47.656943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:47.749952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:47.818028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339f325a-2d14-476c-b1be-a0354cbb7068 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:47.828619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:47.829279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:47.831176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:47.831486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] PATH_INFO: `/v3/projects/5de2f93c9def4c2393a4a775ca9cfb70` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:47.834006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] RBAC: Authorizing `identity:delete_project(project_id=5de2f93c9def4c2393a4a775ca9cfb70)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:47.836444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:47.907324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe4b2ec-7dfe-4d66-b40c-af3957e4c25b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:47.922253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef495818-4f39-434f-acf1-db390aa1ed56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:47.923099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef495818-4f39-434f-acf1-db390aa1ed56 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:47.923384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef495818-4f39-434f-acf1-db390aa1ed56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:47.923656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef495818-4f39-434f-acf1-db390aa1ed56 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:47.925954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef495818-4f39-434f-acf1-db390aa1ed56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:47.950446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef495818-4f39-434f-acf1-db390aa1ed56 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:48.025899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5de2f93c9def4c2393a4a775ca9cfb70', 'request_id': 'req-864a60e3-a254-49d2-9235-b9884da84fbd'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:48.057466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5de2f93c9def4c2393a4a775ca9cfb70', 'request_id': 'req-864a60e3-a254-49d2-9235-b9884da84fbd'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:48.063153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 726/1458] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:47 2022] POST /identity/v3/projects => generated 357 bytes in 413 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:48.064411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef495818-4f39-434f-acf1-db390aa1ed56 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:48.074025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5de2f93c9def4c2393a4a775ca9cfb70', 'request_id': 'req-864a60e3-a254-49d2-9235-b9884da84fbd'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:48.108123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] The token cache is being invalidate because project 5de2f93c9def4c2393a4a775ca9cfb70 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:48.109508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:48.114032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:48.118069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 733/1459] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:47 2022] DELETE /identity/v3/projects/5de2f93c9def4c2393a4a775ca9cfb70 => generated 0 bytes in 466 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:48.118944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:48.128639 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:48.196162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:48.203408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef495818-4f39-434f-acf1-db390aa1ed56 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:48.209242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f789908c-a803-4655-a4b2-1507dcf55cdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:48.210276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f789908c-a803-4655-a4b2-1507dcf55cdb admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:48.210449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f789908c-a803-4655-a4b2-1507dcf55cdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:48.211813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f789908c-a803-4655-a4b2-1507dcf55cdb admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:48.212567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f789908c-a803-4655-a4b2-1507dcf55cdb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:48.218584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f789908c-a803-4655-a4b2-1507dcf55cdb admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:48.264224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 727/1460] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:48 2022] POST /identity/v3/projects => generated 367 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:48.266222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32aae1f-2cf9-47e3-911e-6604a24acbb9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:48.266607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32aae1f-2cf9-47e3-911e-6604a24acbb9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:48.267040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32aae1f-2cf9-47e3-911e-6604a24acbb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:48.281106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864a60e3-a254-49d2-9235-b9884da84fbd admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:48.299593 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a32aae1f-2cf9-47e3-911e-6604a24acbb9 None None] MFA Rules not processed for user `6f8725c4d05b4dd79b05b43fba15980b`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:48.301974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a32aae1f-2cf9-47e3-911e-6604a24acbb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:48.413215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 728/1461] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:48 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:48.413215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32aae1f-2cf9-47e3-911e-6604a24acbb9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:48.416231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a32aae1f-2cf9-47e3-911e-6604a24acbb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:48.428283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d676a72b-15b6-4345-8e42-8cf565cce533 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:48.430868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676a72b-15b6-4345-8e42-8cf565cce533 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:48.432017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676a72b-15b6-4345-8e42-8cf565cce533 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:48.432418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676a72b-15b6-4345-8e42-8cf565cce533 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:48.437974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d676a72b-15b6-4345-8e42-8cf565cce533 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:48.609398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d676a72b-15b6-4345-8e42-8cf565cce533 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:48.613784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d676a72b-15b6-4345-8e42-8cf565cce533 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:48.631567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 734/1462] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:48 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:48.645634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77082455-57b4-4589-a09e-5f5c9a0a73d2 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:48.646397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77082455-57b4-4589-a09e-5f5c9a0a73d2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:48.646397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77082455-57b4-4589-a09e-5f5c9a0a73d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:48.701451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-77082455-57b4-4589-a09e-5f5c9a0a73d2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:48.703525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-77082455-57b4-4589-a09e-5f5c9a0a73d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:48.955137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:48.963738 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 735/1463] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:48.967900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77082455-57b4-4589-a09e-5f5c9a0a73d2 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:48.980910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-77082455-57b4-4589-a09e-5f5c9a0a73d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:49.035692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77082455-57b4-4589-a09e-5f5c9a0a73d2 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:49.045632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32aae1f-2cf9-47e3-911e-6604a24acbb9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:49.061764 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89256956-ec04-42d2-a63b-4468ed8d2450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:49.067257 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89256956-ec04-42d2-a63b-4468ed8d2450 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.068500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89256956-ec04-42d2-a63b-4468ed8d2450 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.071692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89256956-ec04-42d2-a63b-4468ed8d2450 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.073832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89256956-ec04-42d2-a63b-4468ed8d2450 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:49.082378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89256956-ec04-42d2-a63b-4468ed8d2450 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:49.085846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1e934d-a590-4a95-bcb2-f196169cf6d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:49.086536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1e934d-a590-4a95-bcb2-f196169cf6d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.086811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1e934d-a590-4a95-bcb2-f196169cf6d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.087130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1e934d-a590-4a95-bcb2-f196169cf6d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.087958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1e934d-a590-4a95-bcb2-f196169cf6d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:49.095184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1e934d-a590-4a95-bcb2-f196169cf6d2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:49.173397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 729/1464] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:48 2022] POST /identity/v3/users => generated 285 bytes in 765 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:49.179327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754f25e-abfc-4e1c-80fb-15ce951ea9cd None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.179616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754f25e-abfc-4e1c-80fb-15ce951ea9cd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.179892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754f25e-abfc-4e1c-80fb-15ce951ea9cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.200532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 736/1465] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:48 2022] POST /identity/v3/users => generated 290 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:49.211419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50ffbe9-57df-4aaa-9d3d-732da5e8c9c4 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.213574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50ffbe9-57df-4aaa-9d3d-732da5e8c9c4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.214620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50ffbe9-57df-4aaa-9d3d-732da5e8c9c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.283017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f754f25e-abfc-4e1c-80fb-15ce951ea9cd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:49.289190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f754f25e-abfc-4e1c-80fb-15ce951ea9cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:49.311016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 730/1466] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:49.312160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f754f25e-abfc-4e1c-80fb-15ce951ea9cd None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:49.341118 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:49.342688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f754f25e-abfc-4e1c-80fb-15ce951ea9cd None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:49.371700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6906d030-2984-4f39-b8a5-763f5573e47c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:49.372332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6906d030-2984-4f39-b8a5-763f5573e47c admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.372616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6906d030-2984-4f39-b8a5-763f5573e47c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.372890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6906d030-2984-4f39-b8a5-763f5573e47c admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.373552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6906d030-2984-4f39-b8a5-763f5573e47c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:49.383700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6906d030-2984-4f39-b8a5-763f5573e47c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:49.416514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e50ffbe9-57df-4aaa-9d3d-732da5e8c9c4 None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:49.419315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e50ffbe9-57df-4aaa-9d3d-732da5e8c9c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:49.447852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 731/1467] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:49 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:49.450076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6906d030-2984-4f39-b8a5-763f5573e47c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:49.482862 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6906d030-2984-4f39-b8a5-763f5573e47c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:49.500540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4ca232-0fac-48a0-9f05-be6f6d238d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:49.501262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4ca232-0fac-48a0-9f05-be6f6d238d7b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.501788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4ca232-0fac-48a0-9f05-be6f6d238d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.503740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4ca232-0fac-48a0-9f05-be6f6d238d7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.506073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4ca232-0fac-48a0-9f05-be6f6d238d7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:49.510388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4ca232-0fac-48a0-9f05-be6f6d238d7b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:49.548340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 732/1468] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:49 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:49.557713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4ca232-0fac-48a0-9f05-be6f6d238d7b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:49.594582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4ca232-0fac-48a0-9f05-be6f6d238d7b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:49.603026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:49.603671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.603940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.604211 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.604875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:40:49.605190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:49.625877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:49.626032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 737/1469] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:40:49 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 422 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:49.626032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50ffbe9-57df-4aaa-9d3d-732da5e8c9c4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:49.652583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e50ffbe9-57df-4aaa-9d3d-732da5e8c9c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:49.657788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 733/1470] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:40:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:40:49.667067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:49.694771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576d1ead-ee11-40e7-9d6e-f351a55c033b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:49.714894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c74289-3aad-4c49-8fdb-4af31468871c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:49.715595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c74289-3aad-4c49-8fdb-4af31468871c admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.715877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c74289-3aad-4c49-8fdb-4af31468871c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.716152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c74289-3aad-4c49-8fdb-4af31468871c admin admin] PATH_INFO: `/v3/projects/391b03ad8a1a4ebdb2362b59ac01cf78/users/f73c8b191582444d8c7ecf98163faa34/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.746360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50ffbe9-57df-4aaa-9d3d-732da5e8c9c4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:49.749690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c74289-3aad-4c49-8fdb-4af31468871c admin admin] RBAC: Authorizing `identity:create_grant(project_id=391b03ad8a1a4ebdb2362b59ac01cf78, user_id=f73c8b191582444d8c7ecf98163faa34, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:49.752470 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c74289-3aad-4c49-8fdb-4af31468871c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:49.792057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 734/1471] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:49 2022] PUT /identity/v3/projects/391b03ad8a1a4ebdb2362b59ac01cf78/users/f73c8b191582444d8c7ecf98163faa34/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:49.793002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c74289-3aad-4c49-8fdb-4af31468871c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:49.798733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:49.805486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:49.808278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:49.822367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] PATH_INFO: `/v3/users/ce0717e10bb24cbca0b23361e62c1359` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:49.828074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce0717e10bb24cbca0b23361e62c1359)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:49.837576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:50.012858 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c74289-3aad-4c49-8fdb-4af31468871c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:50.039568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:50.040252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.040532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.040806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] PATH_INFO: `/v3/projects/cf905b2711344d459391f06f95b5af8f/users/f7821a9f35d344dab6eb0a7e53beced6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.057706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce0717e10bb24cbca0b23361e62c1359'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:50.084874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce0717e10bb24cbca0b23361e62c1359'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:50.086527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf905b2711344d459391f06f95b5af8f, user_id=f7821a9f35d344dab6eb0a7e53beced6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:50.088626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce0717e10bb24cbca0b23361e62c1359'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:50.089589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce0717e10bb24cbca0b23361e62c1359'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:50.090408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:50.107692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce0717e10bb24cbca0b23361e62c1359'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:50.127796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 735/1472] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:49 2022] PUT /identity/v3/projects/cf905b2711344d459391f06f95b5af8f/users/f7821a9f35d344dab6eb0a7e53beced6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:50.128903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:50.139164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:50.188894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 738/1473] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:49 2022] DELETE /identity/v3/users/ce0717e10bb24cbca0b23361e62c1359 => generated 0 bytes in 564 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:50.190036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:50.201596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:50.317743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:50.370802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f72b3e-7c77-45af-9064-bff3c6f22a0b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:50.389132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:50.389811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.390198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.390458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.391485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:50.393769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:50.427113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a779e1aa-a59e-4280-a26e-dc46fefc8bd6 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:50.438597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:50.452367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.453040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.454046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.456161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:50.458149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:50.461555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:50.477864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 736/1474] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:50 2022] POST /identity/v3/projects => generated 389 bytes in 349 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:50.479388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:50.479746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 739/1475] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:50 2022] GET /identity/v3/roles => generated 1420 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:50.481242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:50.484481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:50.486168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:50.502893 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c2170c-7e5c-4682-ae96-85f037b6b394 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:50.636097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbe2cd0-5c12-4065-8acf-50000ea605ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:50.637764 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe2cd0-5c12-4065-8acf-50000ea605ac admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.639948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe2cd0-5c12-4065-8acf-50000ea605ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.640503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe2cd0-5c12-4065-8acf-50000ea605ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.641647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbe2cd0-5c12-4065-8acf-50000ea605ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:50.645926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbe2cd0-5c12-4065-8acf-50000ea605ac admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:50.670589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe26b47-503e-42a4-bb3c-a9470b422ca7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:50.686585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 740/1476] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:50 2022] GET /identity/v3/roles => generated 1420 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:50.688039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a14a36-222a-400f-841f-63888341c054 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:50.688685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a14a36-222a-400f-841f-63888341c054 service swift] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.688960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a14a36-222a-400f-841f-63888341c054 service swift] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.689226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a14a36-222a-400f-841f-63888341c054 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.691174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-04a14a36-222a-400f-841f-63888341c054 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:50.693010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8b695-51bd-4ed2-9a42-60129a70c4eb None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.693580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8b695-51bd-4ed2-9a42-60129a70c4eb None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.695307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8b695-51bd-4ed2-9a42-60129a70c4eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.755411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-f1c8b695-51bd-4ed2-9a42-60129a70c4eb None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:50.756773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f1c8b695-51bd-4ed2-9a42-60129a70c4eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:50.765295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 741/1477] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:50.766166 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c8b695-51bd-4ed2-9a42-60129a70c4eb None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:50.791891 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c8b695-51bd-4ed2-9a42-60129a70c4eb None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:50.800712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a84bdf-08fd-4e0c-8473-84957a6799cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:50.801503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a84bdf-08fd-4e0c-8473-84957a6799cc admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.801914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a84bdf-08fd-4e0c-8473-84957a6799cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.802366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a84bdf-08fd-4e0c-8473-84957a6799cc admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.803068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a84bdf-08fd-4e0c-8473-84957a6799cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:50.806994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a84bdf-08fd-4e0c-8473-84957a6799cc admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:50.879100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a14a36-222a-400f-841f-63888341c054 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:50.881021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a14a36-222a-400f-841f-63888341c054 service swift] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:50.882391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 742/1478] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:50 2022] POST /identity/v3/users => generated 295 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:50.883264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a84bdf-08fd-4e0c-8473-84957a6799cc admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:50.914756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 737/1479] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:40:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:40:50.915620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a14a36-222a-400f-841f-63888341c054 service swift] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:50.936772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a84bdf-08fd-4e0c-8473-84957a6799cc admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:50.947792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a14a36-222a-400f-841f-63888341c054 service swift] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:50.951274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:50.951811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.952091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.952372 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] PATH_INFO: `/v3/projects/391b03ad8a1a4ebdb2362b59ac01cf78/users/f73c8b191582444d8c7ecf98163faa34/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.961602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=391b03ad8a1a4ebdb2362b59ac01cf78, user_id=f73c8b191582444d8c7ecf98163faa34, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:50.969731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3931cf-2228-4627-ae0c-05d28b8df8df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:50.970716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3931cf-2228-4627-ae0c-05d28b8df8df admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:50.971104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3931cf-2228-4627-ae0c-05d28b8df8df admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:50.971377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3931cf-2228-4627-ae0c-05d28b8df8df admin admin] PATH_INFO: `/v3/projects/cf905b2711344d459391f06f95b5af8f/users/f7821a9f35d344dab6eb0a7e53beced6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:50.971477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:50.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3931cf-2228-4627-ae0c-05d28b8df8df admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf905b2711344d459391f06f95b5af8f, user_id=f7821a9f35d344dab6eb0a7e53beced6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:50.984921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3931cf-2228-4627-ae0c-05d28b8df8df admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:51.017798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 743/1480] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:50 2022] PUT /identity/v3/projects/391b03ad8a1a4ebdb2362b59ac01cf78/users/f73c8b191582444d8c7ecf98163faa34/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:51.017798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:51.017798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:51.037741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 738/1481] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:50 2022] PUT /identity/v3/projects/cf905b2711344d459391f06f95b5af8f/users/f7821a9f35d344dab6eb0a7e53beced6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:51.037741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3931cf-2228-4627-ae0c-05d28b8df8df admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:51.167761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:51.225270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3931cf-2228-4627-ae0c-05d28b8df8df admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:51.235624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7c59da-40bc-4256-a536-6bd3a89fa837 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:51.236251 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c59da-40bc-4256-a536-6bd3a89fa837 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:51.236666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c59da-40bc-4256-a536-6bd3a89fa837 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:51.237004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c59da-40bc-4256-a536-6bd3a89fa837 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:51.237753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7c59da-40bc-4256-a536-6bd3a89fa837 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:51.240644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7c59da-40bc-4256-a536-6bd3a89fa837 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:51.263336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 739/1482] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:51 2022] GET /identity/v3/roles => generated 1420 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:51.264942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8dd44f-33d5-455d-b1a2-26989a6ae127 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:51.265224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8dd44f-33d5-455d-b1a2-26989a6ae127 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:51.265493 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8dd44f-33d5-455d-b1a2-26989a6ae127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:51.312051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2fd752-b9a9-4477-8776-c53ca36a43c9 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:51.325577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ed8dd44f-33d5-455d-b1a2-26989a6ae127 None None] MFA Rules not processed for user `f73c8b191582444d8c7ecf98163faa34`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:51.325577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ed8dd44f-33d5-455d-b1a2-26989a6ae127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:51.594289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 740/1483] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:51 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 327 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:51.594289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5815f493-3a7b-4cde-aba2-7e2b87f05618 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:51.594289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5815f493-3a7b-4cde-aba2-7e2b87f05618 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:51.594289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5815f493-3a7b-4cde-aba2-7e2b87f05618 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:51.607440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d286a23b-7f03-4c40-8ec9-6508b6c6933f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:51.609513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d286a23b-7f03-4c40-8ec9-6508b6c6933f service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:51.611058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d286a23b-7f03-4c40-8ec9-6508b6c6933f service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:51.612824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d286a23b-7f03-4c40-8ec9-6508b6c6933f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:51.652539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d286a23b-7f03-4c40-8ec9-6508b6c6933f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:51.653192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5815f493-3a7b-4cde-aba2-7e2b87f05618 None None] MFA Rules not processed for user `f7821a9f35d344dab6eb0a7e53beced6`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:51.653470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d286a23b-7f03-4c40-8ec9-6508b6c6933f service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:51.654546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5815f493-3a7b-4cde-aba2-7e2b87f05618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:51.682591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 744/1484] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 673 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:51.683532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d286a23b-7f03-4c40-8ec9-6508b6c6933f service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:51.717817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d286a23b-7f03-4c40-8ec9-6508b6c6933f service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:51.740604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:51.742509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:51.742509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:51.742709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:51.749147 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:51.756505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 741/1485] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:51 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:51.756505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5815f493-3a7b-4cde-aba2-7e2b87f05618 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:51.756505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5815f493-3a7b-4cde-aba2-7e2b87f05618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:51.756831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:51.795990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 745/1486] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:51 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:51.796902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:51.811818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:51.855318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:52.042905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5815f493-3a7b-4cde-aba2-7e2b87f05618 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:52.062101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b31178-b862-4318-8ee0-e4dceb7c41a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:52.063949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b31178-b862-4318-8ee0-e4dceb7c41a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:52.066589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b31178-b862-4318-8ee0-e4dceb7c41a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:52.067091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b31178-b862-4318-8ee0-e4dceb7c41a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:52.091335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:52.107683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b31178-b862-4318-8ee0-e4dceb7c41a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:52.123731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b31178-b862-4318-8ee0-e4dceb7c41a7 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:52.128099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25365cf6-8fd9-4da9-9fec-c2871ef3e2aa admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:52.140025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bea14c-7df0-46a6-8ea4-2c98d720bad1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:52.140721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bea14c-7df0-46a6-8ea4-2c98d720bad1 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:52.140996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bea14c-7df0-46a6-8ea4-2c98d720bad1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:52.141264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bea14c-7df0-46a6-8ea4-2c98d720bad1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:52.169086 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 742/1487] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:51 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:52.173097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b31178-b862-4318-8ee0-e4dceb7c41a7 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:52.216297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b31178-b862-4318-8ee0-e4dceb7c41a7 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:52.234824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bea14c-7df0-46a6-8ea4-2c98d720bad1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:52.239310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bea14c-7df0-46a6-8ea4-2c98d720bad1 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:52.254039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c37600f-6b45-4eb2-8da7-11a1d13e574f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:52.255982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c37600f-6b45-4eb2-8da7-11a1d13e574f admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:52.259797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c37600f-6b45-4eb2-8da7-11a1d13e574f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:52.261021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c37600f-6b45-4eb2-8da7-11a1d13e574f admin admin] PATH_INFO: `/v3/projects/5c653c4c50a54094b19164cf416fd8b4/users/0e32ee2be68f4157a9261b7dfc173d53/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:52.319868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 746/1488] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:51 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:52.372165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c37600f-6b45-4eb2-8da7-11a1d13e574f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c653c4c50a54094b19164cf416fd8b4, user_id=0e32ee2be68f4157a9261b7dfc173d53, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:52.384094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c37600f-6b45-4eb2-8da7-11a1d13e574f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:52.435889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 743/1489] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:52 2022] PUT /identity/v3/projects/5c653c4c50a54094b19164cf416fd8b4/users/0e32ee2be68f4157a9261b7dfc173d53/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:52.446577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1288982-ddef-488c-b484-387388680225 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:52.448078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1288982-ddef-488c-b484-387388680225 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:52.450016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1288982-ddef-488c-b484-387388680225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:52.498576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c37600f-6b45-4eb2-8da7-11a1d13e574f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:52.533090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-b1288982-ddef-488c-b484-387388680225 None None] MFA Rules not processed for user `0e32ee2be68f4157a9261b7dfc173d53`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:52.536091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b1288982-ddef-488c-b484-387388680225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:52.639092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c37600f-6b45-4eb2-8da7-11a1d13e574f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:52.643536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:52.643837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:52.644122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:52.644400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:52.660566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:52.818588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 747/1490] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:52 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 370 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:52.821859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1288982-ddef-488c-b484-387388680225 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:52.858135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1288982-ddef-488c-b484-387388680225 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:52.883150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8428dea4-3688-478b-b40c-f0a81b156fa8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:52.884715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428dea4-3688-478b-b40c-f0a81b156fa8 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:52.885390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428dea4-3688-478b-b40c-f0a81b156fa8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:52.885793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428dea4-3688-478b-b40c-f0a81b156fa8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:52.898185 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8428dea4-3688-478b-b40c-f0a81b156fa8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:52.935169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:52.936697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:52.995646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 744/1491] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:52 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:52.996771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:53.160521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8428dea4-3688-478b-b40c-f0a81b156fa8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:53.177231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8428dea4-3688-478b-b40c-f0a81b156fa8 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:53.230782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 748/1492] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:52 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:53.240604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fed614-49c6-4375-9537-689c80478a9a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:53.242199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fed614-49c6-4375-9537-689c80478a9a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:53.243869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fed614-49c6-4375-9537-689c80478a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:53.296447 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eabd587-1c1a-457f-9eab-72a39a317873 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:53.329506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90f5be6-0d44-4753-b2f2-7090546f2e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:53.330316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90f5be6-0d44-4753-b2f2-7090546f2e69 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:53.330771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90f5be6-0d44-4753-b2f2-7090546f2e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:53.331055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90f5be6-0d44-4753-b2f2-7090546f2e69 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:53.331778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90f5be6-0d44-4753-b2f2-7090546f2e69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:53.339107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90f5be6-0d44-4753-b2f2-7090546f2e69 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:53.375639 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d3fed614-49c6-4375-9537-689c80478a9a None None] MFA Rules not processed for user `5756f4ff604a413b8af1f12c82cc4870`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:53.387163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d3fed614-49c6-4375-9537-689c80478a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:53.468660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 745/1493] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:40:52 2022] POST /identity/v3/projects => generated 389 bytes in 472 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:53.469682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90f5be6-0d44-4753-b2f2-7090546f2e69 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:53.530574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:53.617977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:53.709008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90f5be6-0d44-4753-b2f2-7090546f2e69 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:53.716836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 749/1494] 10.222.0.20 () {58 vars in 1015 bytes} [Wed Oct 19 00:40:53 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 485 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:53.725242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4786f-822b-4d6f-b805-4bc2559b8e9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:53.725972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4786f-822b-4d6f-b805-4bc2559b8e9e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:53.726176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4786f-822b-4d6f-b805-4bc2559b8e9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:53.726405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf4786f-822b-4d6f-b805-4bc2559b8e9e admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:53.727073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf4786f-822b-4d6f-b805-4bc2559b8e9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:53.730938 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf4786f-822b-4d6f-b805-4bc2559b8e9e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:53.777337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fed614-49c6-4375-9537-689c80478a9a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:53.791246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d3fed614-49c6-4375-9537-689c80478a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:53.811159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fed614-49c6-4375-9537-689c80478a9a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:53.812037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 746/1495] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:40:53 2022] POST /identity/v3/users => generated 301 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:40:53.813131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4786f-822b-4d6f-b805-4bc2559b8e9e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:53.847297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf4786f-822b-4d6f-b805-4bc2559b8e9e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:53.866161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576f9817-cec9-4a2c-ab42-e46752e7bdd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:53.866788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f9817-cec9-4a2c-ab42-e46752e7bdd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:53.867066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f9817-cec9-4a2c-ab42-e46752e7bdd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:53.867332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f9817-cec9-4a2c-ab42-e46752e7bdd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:53.867981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576f9817-cec9-4a2c-ab42-e46752e7bdd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:53.869891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576f9817-cec9-4a2c-ab42-e46752e7bdd8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:53.903532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 747/1496] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:53 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:53.904525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576f9817-cec9-4a2c-ab42-e46752e7bdd8 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:53.934732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576f9817-cec9-4a2c-ab42-e46752e7bdd8 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:53.959610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf71db-9b8c-4749-b33b-6405604996aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:53.960301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf71db-9b8c-4749-b33b-6405604996aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:53.960577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf71db-9b8c-4749-b33b-6405604996aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:53.960845 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf71db-9b8c-4749-b33b-6405604996aa admin admin] PATH_INFO: `/v3/projects/98bd761d464f48b1b62ba550909c13d3/users/91af21a488d444798eaa0adee9ebb947/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:54.022264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf71db-9b8c-4749-b33b-6405604996aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=98bd761d464f48b1b62ba550909c13d3, user_id=91af21a488d444798eaa0adee9ebb947, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:54.026127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf71db-9b8c-4749-b33b-6405604996aa admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:54.075274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1073b76d-bd61-4db1-9d89-1865ff1937a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:54.075922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1073b76d-bd61-4db1-9d89-1865ff1937a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:54.076200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1073b76d-bd61-4db1-9d89-1865ff1937a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:54.076526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1073b76d-bd61-4db1-9d89-1865ff1937a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:54.079714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 748/1497] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:53 2022] PUT /identity/v3/projects/98bd761d464f48b1b62ba550909c13d3/users/91af21a488d444798eaa0adee9ebb947/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:54.080726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf71db-9b8c-4749-b33b-6405604996aa admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:54.098360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1073b76d-bd61-4db1-9d89-1865ff1937a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:54.100717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1073b76d-bd61-4db1-9d89-1865ff1937a7 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:54.140698 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 750/1498] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:40:54.417282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf71db-9b8c-4749-b33b-6405604996aa admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:54.437869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf58c7-adb6-4a56-a09c-179c4f2e662e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:54.440820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf58c7-adb6-4a56-a09c-179c4f2e662e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:54.444575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf58c7-adb6-4a56-a09c-179c4f2e662e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:54.445642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf58c7-adb6-4a56-a09c-179c4f2e662e admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:54.450063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cf58c7-adb6-4a56-a09c-179c4f2e662e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:54.466617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cf58c7-adb6-4a56-a09c-179c4f2e662e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:54.560326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 749/1499] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:40:54 2022] GET /identity/v3/roles => generated 1420 bytes in 480 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:40:54.566524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1073b76d-bd61-4db1-9d89-1865ff1937a7 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:54.617113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1073b76d-bd61-4db1-9d89-1865ff1937a7 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:54.651503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f276b0b-b877-4c98-bd5f-51c4798a774f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:54.652397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f276b0b-b877-4c98-bd5f-51c4798a774f admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:54.652773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f276b0b-b877-4c98-bd5f-51c4798a774f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:54.653148 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f276b0b-b877-4c98-bd5f-51c4798a774f admin admin] PATH_INFO: `/v3/projects/98bd761d464f48b1b62ba550909c13d3/users/91af21a488d444798eaa0adee9ebb947/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:54.667578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f276b0b-b877-4c98-bd5f-51c4798a774f admin admin] RBAC: Authorizing `identity:create_grant(project_id=98bd761d464f48b1b62ba550909c13d3, user_id=91af21a488d444798eaa0adee9ebb947, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:54.670925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f276b0b-b877-4c98-bd5f-51c4798a774f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:54.737840 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 751/1500] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:40:54 2022] PUT /identity/v3/projects/98bd761d464f48b1b62ba550909c13d3/users/91af21a488d444798eaa0adee9ebb947/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:54.749386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc7973-4470-43fe-9d02-f8f761e65047 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:54.749718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc7973-4470-43fe-9d02-f8f761e65047 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:54.749971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc7973-4470-43fe-9d02-f8f761e65047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:54.862846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1dcc7973-4470-43fe-9d02-f8f761e65047 None None] MFA Rules not processed for user `91af21a488d444798eaa0adee9ebb947`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:54.872615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1dcc7973-4470-43fe-9d02-f8f761e65047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:55.186509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:55.238693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 750/1501] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:54 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 490 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:55.257296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f276b0b-b877-4c98-bd5f-51c4798a774f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:55.475612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f276b0b-b877-4c98-bd5f-51c4798a774f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:55.485631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:55.486662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:55.487106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:55.487614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:55.541662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:55.552671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:55.595211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 752/1502] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:55 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:40:56.035402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc7973-4470-43fe-9d02-f8f761e65047 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:56.225956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:56.227791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:56.268029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc7973-4470-43fe-9d02-f8f761e65047 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:56.284763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:56.285300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:56.285667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:56.285917 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] PATH_INFO: `/v3/users/0e32ee2be68f4157a9261b7dfc173d53` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:56.289293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e32ee2be68f4157a9261b7dfc173d53)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:56.291276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:56.383862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b860ae-6e58-4bb9-9713-9243c3dc86e6 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:56.390081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e32ee2be68f4157a9261b7dfc173d53'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.402466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:56.403442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:56.403884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:56.404424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] PATH_INFO: `/v3/projects/326db2eede964aec9dbd06693dd47cb1` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:56.406800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=326db2eede964aec9dbd06693dd47cb1)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:56.407582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e32ee2be68f4157a9261b7dfc173d53'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.411134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e32ee2be68f4157a9261b7dfc173d53'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.412212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e32ee2be68f4157a9261b7dfc173d53'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.412766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:56.428234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2e7b046f-2131-4f07-8220-9ea28b2df726 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e32ee2be68f4157a9261b7dfc173d53'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.459747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:56.516525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 751/1503] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:56 2022] DELETE /identity/v3/users/0e32ee2be68f4157a9261b7dfc173d53 => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:56.518941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da76d3ac-53aa-473e-8be2-3156f191c4ca None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:56.519232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da76d3ac-53aa-473e-8be2-3156f191c4ca None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:56.520297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da76d3ac-53aa-473e-8be2-3156f191c4ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:56.568567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '326db2eede964aec9dbd06693dd47cb1', 'request_id': 'req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.581617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '326db2eede964aec9dbd06693dd47cb1', 'request_id': 'req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.589321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-da76d3ac-53aa-473e-8be2-3156f191c4ca None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:40:56.590280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '326db2eede964aec9dbd06693dd47cb1', 'request_id': 'req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.598303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-da76d3ac-53aa-473e-8be2-3156f191c4ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:56.626690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] The token cache is being invalidate because project 326db2eede964aec9dbd06693dd47cb1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:40:56.629008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.631517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:56.634782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 753/1504] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:40:56 2022] DELETE /identity/v3/projects/326db2eede964aec9dbd06693dd47cb1 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:40:56.638745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:56.641207 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:56.796021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 752/1505] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:40:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:40:56.818259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a526aa-d98d-4d67-91fd-e2962a42e5ae admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:56.823542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:56.824066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:56.824351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:56.824624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] PATH_INFO: `/v3/users/3804dc9cbb1c4661b58840115d305942` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:56.825812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3804dc9cbb1c4661b58840115d305942)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:56.827843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:57.029973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da76d3ac-53aa-473e-8be2-3156f191c4ca None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:57.031567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-da76d3ac-53aa-473e-8be2-3156f191c4ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:57.040151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:57.164382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3804dc9cbb1c4661b58840115d305942'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:57.196644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3804dc9cbb1c4661b58840115d305942'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:57.198319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3804dc9cbb1c4661b58840115d305942'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:57.203029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3804dc9cbb1c4661b58840115d305942'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:57.234126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3804dc9cbb1c4661b58840115d305942'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:40:57.275765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da76d3ac-53aa-473e-8be2-3156f191c4ca None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:57.287807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5c0184-e581-46f4-89d9-25377d2fe453 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:57.288879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5c0184-e581-46f4-89d9-25377d2fe453 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:57.289321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5c0184-e581-46f4-89d9-25377d2fe453 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:57.289750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5c0184-e581-46f4-89d9-25377d2fe453 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:57.300928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4a5c0184-e581-46f4-89d9-25377d2fe453 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:57.319987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 754/1506] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:56 2022] DELETE /identity/v3/users/3804dc9cbb1c4661b58840115d305942 => generated 0 bytes in 679 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:40:57.392893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5c0184-e581-46f4-89d9-25377d2fe453 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:57.394810 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5c0184-e581-46f4-89d9-25377d2fe453 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:40:57.418326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 753/1507] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:40:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:40:59.674797 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:40:59.677882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:40:59.723837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:40:59.852410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070c97ef-60ca-4313-a62f-eaa82e93788a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:40:59.866225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:40:59.867799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:40:59.868465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:40:59.869116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] PATH_INFO: `/v3/users/13146d4ec6fa40a2ad1c8ee0647199bf` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:40:59.879835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] RBAC: Authorizing `identity:delete_user(user_id=13146d4ec6fa40a2ad1c8ee0647199bf)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:40:59.894256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:00.062584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13146d4ec6fa40a2ad1c8ee0647199bf'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.092366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13146d4ec6fa40a2ad1c8ee0647199bf'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.094552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13146d4ec6fa40a2ad1c8ee0647199bf'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.096646 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13146d4ec6fa40a2ad1c8ee0647199bf'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.122454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13146d4ec6fa40a2ad1c8ee0647199bf'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.174646 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 755/1508] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:40:59 2022] DELETE /identity/v3/users/13146d4ec6fa40a2ad1c8ee0647199bf => generated 0 bytes in 498 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:00.176647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1c98d-ba6c-42c9-8ad2-40ccc0ff7306 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:00.177203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1c98d-ba6c-42c9-8ad2-40ccc0ff7306 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:00.179244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc1c98d-ba6c-42c9-8ad2-40ccc0ff7306 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:00.228890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-dfc1c98d-ba6c-42c9-8ad2-40ccc0ff7306 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:00.232832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-dfc1c98d-ba6c-42c9-8ad2-40ccc0ff7306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:00.372026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 754/1509] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:00.500130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:00.502464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:00.628771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc1c98d-ba6c-42c9-8ad2-40ccc0ff7306 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:00.637697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-dfc1c98d-ba6c-42c9-8ad2-40ccc0ff7306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:00.677622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc1c98d-ba6c-42c9-8ad2-40ccc0ff7306 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:00.690055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:00.694961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:00.697277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:00.697869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] PATH_INFO: `/v3/projects/7317ead796c04b7593525ce05e02505c` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:00.699928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7317ead796c04b7593525ce05e02505c)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:00.726795 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:00.759904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b19c7-4a51-456d-ab44-f9f0b4afa60d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:00.780248 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:00.780889 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:00.781168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:00.781444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:00.799229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:00.803520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:00.827537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 756/1510] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:00.831476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:00.866564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:00.898698 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6704022a-db0e-4ad0-b062-b6854d5caa15 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:00.906927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7317ead796c04b7593525ce05e02505c', 'request_id': 'req-623fe384-cd9b-4888-8dee-df232e3a8e65'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.933324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7317ead796c04b7593525ce05e02505c', 'request_id': 'req-623fe384-cd9b-4888-8dee-df232e3a8e65'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.944434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7317ead796c04b7593525ce05e02505c', 'request_id': 'req-623fe384-cd9b-4888-8dee-df232e3a8e65'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.977830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] The token cache is being invalidate because project 7317ead796c04b7593525ce05e02505c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:00.978356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.980329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-623fe384-cd9b-4888-8dee-df232e3a8e65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:00.988569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 755/1511] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:00 2022] DELETE /identity/v3/projects/7317ead796c04b7593525ce05e02505c => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:00.990733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f55e78-3711-4ad7-8908-996c8cfc66d6 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:00.992092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f55e78-3711-4ad7-8908-996c8cfc66d6 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:00.992731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f55e78-3711-4ad7-8908-996c8cfc66d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.002835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:01.030334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b6f55e78-3711-4ad7-8908-996c8cfc66d6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:01.031996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b6f55e78-3711-4ad7-8908-996c8cfc66d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:01.033970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:01.034749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:01.035032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:01.035468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] PATH_INFO: `/v3/projects/5c653c4c50a54094b19164cf416fd8b4` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.051813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c653c4c50a54094b19164cf416fd8b4)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:01.051813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:01.053514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 756/1512] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:01.055882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c944292-8fb1-46dd-9586-89b70dda5b9a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:01.056431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c944292-8fb1-46dd-9586-89b70dda5b9a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:01.056909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c944292-8fb1-46dd-9586-89b70dda5b9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.087303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5c944292-8fb1-46dd-9586-89b70dda5b9a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:01.089089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5c944292-8fb1-46dd-9586-89b70dda5b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:01.122830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 757/1513] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:01.134551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c944292-8fb1-46dd-9586-89b70dda5b9a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:01.150085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c944292-8fb1-46dd-9586-89b70dda5b9a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:01.170661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:01.214423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c653c4c50a54094b19164cf416fd8b4', 'request_id': 'req-71e86ce1-243c-4e90-90d1-acbf6f237895'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.256156 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c653c4c50a54094b19164cf416fd8b4', 'request_id': 'req-71e86ce1-243c-4e90-90d1-acbf6f237895'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.271959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:01.273239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:01.273770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:01.274851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.275321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:01.275826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:01.279442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:01.301025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 758/1514] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:01.302366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:01.318665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dbf9de-e56e-4623-993a-c8e68a87b541 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:01.319291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c653c4c50a54094b19164cf416fd8b4', 'request_id': 'req-71e86ce1-243c-4e90-90d1-acbf6f237895'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.341179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] The token cache is being invalidate because project 5c653c4c50a54094b19164cf416fd8b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:01.341479 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.343819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662aa757-ea41-487a-826d-a38512d3f66d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:01.344989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662aa757-ea41-487a-826d-a38512d3f66d admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:01.346422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662aa757-ea41-487a-826d-a38512d3f66d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:01.346935 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.347369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662aa757-ea41-487a-826d-a38512d3f66d admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.349891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662aa757-ea41-487a-826d-a38512d3f66d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:01.350973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 757/1515] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:00 2022] DELETE /identity/v3/projects/5c653c4c50a54094b19164cf416fd8b4 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:01.355068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:01.355323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662aa757-ea41-487a-826d-a38512d3f66d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:01.360604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:01.394085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e86ce1-243c-4e90-90d1-acbf6f237895 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:01.413020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:01.413677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:01.413956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:01.414236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] PATH_INFO: `/v3/users/91af21a488d444798eaa0adee9ebb947` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.415457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] RBAC: Authorizing `identity:delete_user(user_id=91af21a488d444798eaa0adee9ebb947)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:01.417413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:01.500212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 759/1516] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:01 2022] POST /identity/v3/projects => generated 373 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:01.503979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a3af15-e9aa-4c46-8df4-982c06619932 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:01.504477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a3af15-e9aa-4c46-8df4-982c06619932 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:01.504897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a3af15-e9aa-4c46-8df4-982c06619932 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.523568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 760/1517] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:41:01 2022] GET /identity => generated 270 bytes in 22 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:41:01.526704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a3af15-e9aa-4c46-8df4-982c06619932 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:01.535046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-05a3af15-e9aa-4c46-8df4-982c06619932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:01.552025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91af21a488d444798eaa0adee9ebb947'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.576502 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91af21a488d444798eaa0adee9ebb947'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.577264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91af21a488d444798eaa0adee9ebb947'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.577902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91af21a488d444798eaa0adee9ebb947'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.593710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a3393a1f-a16d-43ed-a8f7-e430950b497a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91af21a488d444798eaa0adee9ebb947'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:01.627779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 758/1518] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:01 2022] DELETE /identity/v3/users/91af21a488d444798eaa0adee9ebb947 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:01.628431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea9eb5-5d0b-4ff1-8a2f-489c7fb22ebf None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:01.629127 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea9eb5-5d0b-4ff1-8a2f-489c7fb22ebf None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:01.629535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea9eb5-5d0b-4ff1-8a2f-489c7fb22ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.673427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-71ea9eb5-5d0b-4ff1-8a2f-489c7fb22ebf None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:01.677079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-71ea9eb5-5d0b-4ff1-8a2f-489c7fb22ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:01.719365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a3af15-e9aa-4c46-8df4-982c06619932 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:01.845297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:01.917306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 759/1519] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:41:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:01.936219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0a7afb-8d9b-4e4d-8f88-62853549d74d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:01.937817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0a7afb-8d9b-4e4d-8f88-62853549d74d admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:01.945286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0a7afb-8d9b-4e4d-8f88-62853549d74d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:01.945286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0a7afb-8d9b-4e4d-8f88-62853549d74d admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:01.945286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0a7afb-8d9b-4e4d-8f88-62853549d74d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:01.945286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0a7afb-8d9b-4e4d-8f88-62853549d74d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:02.002535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ea9eb5-5d0b-4ff1-8a2f-489c7fb22ebf None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:02.002814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 761/1520] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:01 2022] POST /identity/v3/users => generated 287 bytes in 478 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:02.004538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-71ea9eb5-5d0b-4ff1-8a2f-489c7fb22ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:02.037542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ea9eb5-5d0b-4ff1-8a2f-489c7fb22ebf None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:02.049322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8c4e2-436a-4a3a-9c2d-4bd89912ac3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:02.050442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8c4e2-436a-4a3a-9c2d-4bd89912ac3a admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:02.050858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8c4e2-436a-4a3a-9c2d-4bd89912ac3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:02.051241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8c4e2-436a-4a3a-9c2d-4bd89912ac3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:02.053217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b8c4e2-436a-4a3a-9c2d-4bd89912ac3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:02.056822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b8c4e2-436a-4a3a-9c2d-4bd89912ac3a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:02.109725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 760/1521] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:02 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:02.113927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0a7afb-8d9b-4e4d-8f88-62853549d74d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:02.152347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0a7afb-8d9b-4e4d-8f88-62853549d74d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:02.162335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83bd400-1ff3-411e-aeac-41cb9a69ced3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:02.165467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83bd400-1ff3-411e-aeac-41cb9a69ced3 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:02.165923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83bd400-1ff3-411e-aeac-41cb9a69ced3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:02.166403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83bd400-1ff3-411e-aeac-41cb9a69ced3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:02.167275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83bd400-1ff3-411e-aeac-41cb9a69ced3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:02.170657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83bd400-1ff3-411e-aeac-41cb9a69ced3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:02.191350 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8c4e2-436a-4a3a-9c2d-4bd89912ac3a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:02.198893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 762/1522] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:02 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:02.207304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b8c4e2-436a-4a3a-9c2d-4bd89912ac3a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:02.223584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:02.224739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:02.226163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:02.226816 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] PATH_INFO: `/v3/projects/0e082898df784f5c8eaa55f0cc717de5/users/0e0407fdc07f4ad98d09fdabbc9952fa/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:02.272245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e082898df784f5c8eaa55f0cc717de5, user_id=0e0407fdc07f4ad98d09fdabbc9952fa, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:02.275638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:02.305896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ced2400-cbcc-44b0-9dca-9b73286721a4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:02.307077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ced2400-cbcc-44b0-9dca-9b73286721a4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:02.307732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ced2400-cbcc-44b0-9dca-9b73286721a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:02.312998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 761/1523] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:02 2022] PUT /identity/v3/projects/0e082898df784f5c8eaa55f0cc717de5/users/0e0407fdc07f4ad98d09fdabbc9952fa/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:02.361630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-7ced2400-cbcc-44b0-9dca-9b73286721a4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:02.365555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7ced2400-cbcc-44b0-9dca-9b73286721a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:02.400066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:02.610632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:02.615088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:02.685491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 763/1524] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:02.713499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ced2400-cbcc-44b0-9dca-9b73286721a4 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:02.715483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7ced2400-cbcc-44b0-9dca-9b73286721a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:02.926539 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ced2400-cbcc-44b0-9dca-9b73286721a4 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:02.930014 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0b127f-47ad-48ba-b285-7545b0178be5 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:02.940043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab4c96-1a88-4b38-942e-aeb5a41a7337 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:02.942105 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab4c96-1a88-4b38-942e-aeb5a41a7337 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:02.942457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab4c96-1a88-4b38-942e-aeb5a41a7337 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:02.942719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab4c96-1a88-4b38-942e-aeb5a41a7337 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:02.943696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08b5480-eab4-4ae9-84b9-ead5a4376bdf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:02.944484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08b5480-eab4-4ae9-84b9-ead5a4376bdf service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:02.944610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08b5480-eab4-4ae9-84b9-ead5a4376bdf service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:02.944859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08b5480-eab4-4ae9-84b9-ead5a4376bdf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:02.954957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab4c96-1a88-4b38-942e-aeb5a41a7337 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:02.957089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab4c96-1a88-4b38-942e-aeb5a41a7337 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:02.966731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08b5480-eab4-4ae9-84b9-ead5a4376bdf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:02.968284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08b5480-eab4-4ae9-84b9-ead5a4376bdf service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:02.970965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 764/1525] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:02.992074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 762/1526] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:03.507173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691f5905-633e-436f-a7e6-df03c4ead08c None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:03.507812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691f5905-633e-436f-a7e6-df03c4ead08c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:03.508092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691f5905-633e-436f-a7e6-df03c4ead08c None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:03.510443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 765/1527] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:41:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:41:03.657596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fd370-3b78-4568-b681-ce4a44c92aae None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:03.659650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fd370-3b78-4568-b681-ce4a44c92aae None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:03.659650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3fd370-3b78-4568-b681-ce4a44c92aae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:03.707844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ab3fd370-3b78-4568-b681-ce4a44c92aae None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:03.709101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ab3fd370-3b78-4568-b681-ce4a44c92aae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:03.730364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 763/1528] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:41:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:04.040409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691f5905-633e-436f-a7e6-df03c4ead08c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:04.042094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-691f5905-633e-436f-a7e6-df03c4ead08c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:04.075817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691f5905-633e-436f-a7e6-df03c4ead08c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:04.081943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:04.082875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:04.082965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:04.083240 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] PATH_INFO: `/v3/projects/5e9336eec0d3469ebb7ba20cb4fe8f3c` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:04.084477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5e9336eec0d3469ebb7ba20cb4fe8f3c)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:04.086413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:04.196924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5e9336eec0d3469ebb7ba20cb4fe8f3c', 'request_id': 'req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.213130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5e9336eec0d3469ebb7ba20cb4fe8f3c', 'request_id': 'req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.227620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5e9336eec0d3469ebb7ba20cb4fe8f3c', 'request_id': 'req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.242047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] The token cache is being invalidate because project 5e9336eec0d3469ebb7ba20cb4fe8f3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:04.242912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.243849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.245652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 766/1529] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:04 2022] DELETE /identity/v3/projects/5e9336eec0d3469ebb7ba20cb4fe8f3c => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:04.249858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3fd370-3b78-4568-b681-ce4a44c92aae None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:04.275883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3fd370-3b78-4568-b681-ce4a44c92aae None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:04.371997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:04.372959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:04.373770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:04.374205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] PATH_INFO: `/v3/users/f7821a9f35d344dab6eb0a7e53beced6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:04.376997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7821a9f35d344dab6eb0a7e53beced6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:04.380066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:04.413779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:04.415428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:04.456164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:04.492536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7821a9f35d344dab6eb0a7e53beced6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.503189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7821a9f35d344dab6eb0a7e53beced6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.504004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7821a9f35d344dab6eb0a7e53beced6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.504697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7821a9f35d344dab6eb0a7e53beced6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.515938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7821a9f35d344dab6eb0a7e53beced6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.527751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:04.554951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 764/1530] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:04 2022] DELETE /identity/v3/users/f7821a9f35d344dab6eb0a7e53beced6 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:04.554951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:04.554951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:04.596579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665b6df-1b3d-4fdf-893b-ab5ae02d6150 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:04.752924 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8265a7-fa0f-4784-b55b-071365c8052e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:04.759859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:04.760601 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:04.760945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:04.761283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] PATH_INFO: `/v3/projects/98bd761d464f48b1b62ba550909c13d3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:04.762523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=98bd761d464f48b1b62ba550909c13d3)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:04.764803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:04.791283 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120c2255-4947-42fe-9f90-de27aad788af service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:04.792073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120c2255-4947-42fe-9f90-de27aad788af service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:04.792364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120c2255-4947-42fe-9f90-de27aad788af service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:04.792660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120c2255-4947-42fe-9f90-de27aad788af service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:04.794785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-120c2255-4947-42fe-9f90-de27aad788af service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:04.814787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120c2255-4947-42fe-9f90-de27aad788af service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:04.818671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120c2255-4947-42fe-9f90-de27aad788af service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:04.831184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98bd761d464f48b1b62ba550909c13d3', 'request_id': 'req-5417e2fb-befb-43d5-b966-02dfff8c34ed'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.843056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98bd761d464f48b1b62ba550909c13d3', 'request_id': 'req-5417e2fb-befb-43d5-b966-02dfff8c34ed'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.850969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 767/1531] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:04.854899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98bd761d464f48b1b62ba550909c13d3', 'request_id': 'req-5417e2fb-befb-43d5-b966-02dfff8c34ed'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.887713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] The token cache is being invalidate because project 98bd761d464f48b1b62ba550909c13d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:04.891018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5417e2fb-befb-43d5-b966-02dfff8c34ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:04.897900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 765/1532] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:04 2022] DELETE /identity/v3/projects/98bd761d464f48b1b62ba550909c13d3 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:04.902897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb97ce8-03e8-47d7-9faa-2a0aec0584fa None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:04.903329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb97ce8-03e8-47d7-9faa-2a0aec0584fa None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:04.903433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb97ce8-03e8-47d7-9faa-2a0aec0584fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:04.961527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-bdb97ce8-03e8-47d7-9faa-2a0aec0584fa None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:04.965308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-bdb97ce8-03e8-47d7-9faa-2a0aec0584fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:05.081967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 768/1533] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:05.089806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae9a413-2de6-4b44-80cc-d25285611e58 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:05.090117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae9a413-2de6-4b44-80cc-d25285611e58 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:05.090662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae9a413-2de6-4b44-80cc-d25285611e58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:05.196296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-aae9a413-2de6-4b44-80cc-d25285611e58 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:05.199353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-aae9a413-2de6-4b44-80cc-d25285611e58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:05.236148 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 766/1534] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:05.238079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb97ce8-03e8-47d7-9faa-2a0aec0584fa None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:05.252384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb97ce8-03e8-47d7-9faa-2a0aec0584fa None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:05.267782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:05.268503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:05.268704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:05.268982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:05.269646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:05.269953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:05.271848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:05.290212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 769/1535] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:05.291416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae9a413-2de6-4b44-80cc-d25285611e58 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:05.304198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae9a413-2de6-4b44-80cc-d25285611e58 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:05.311894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb9055a-5d4a-415d-a8d0-a6aff4a95fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:05.312636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb9055a-5d4a-415d-a8d0-a6aff4a95fcc admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:05.313268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb9055a-5d4a-415d-a8d0-a6aff4a95fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:05.313268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb9055a-5d4a-415d-a8d0-a6aff4a95fcc admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:05.314778 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb9055a-5d4a-415d-a8d0-a6aff4a95fcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:05.317573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb9055a-5d4a-415d-a8d0-a6aff4a95fcc admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:05.353804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 767/1536] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:05 2022] POST /identity/v3/projects => generated 385 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:05.357725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:05.374170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193434e9-7bb6-4325-8561-17b543d79924 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:05.477869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:05.510544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e1503d-8efc-4676-bd84-fb202ce9032e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:05.510544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e1503d-8efc-4676-bd84-fb202ce9032e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:05.510544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e1503d-8efc-4676-bd84-fb202ce9032e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:05.510544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e1503d-8efc-4676-bd84-fb202ce9032e admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:05.510970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e1503d-8efc-4676-bd84-fb202ce9032e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:05.510970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e1503d-8efc-4676-bd84-fb202ce9032e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:05.560464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 770/1537] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:05 2022] POST /identity/v3/users => generated 293 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:05.564951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb9055a-5d4a-415d-a8d0-a6aff4a95fcc admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:05.575982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb9055a-5d4a-415d-a8d0-a6aff4a95fcc admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:05.584429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24087514-ca98-4e05-8cc6-fcf71badbe23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:05.587464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24087514-ca98-4e05-8cc6-fcf71badbe23 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:05.588745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24087514-ca98-4e05-8cc6-fcf71badbe23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:05.589697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24087514-ca98-4e05-8cc6-fcf71badbe23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:05.591302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24087514-ca98-4e05-8cc6-fcf71badbe23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:05.594513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24087514-ca98-4e05-8cc6-fcf71badbe23 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:05.597586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:05.615386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 768/1538] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:05 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:05.616664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e1503d-8efc-4676-bd84-fb202ce9032e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:05.636233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e1503d-8efc-4676-bd84-fb202ce9032e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:05.641045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87077187-d3cd-483f-af82-eaea36426618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:05.641888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87077187-d3cd-483f-af82-eaea36426618 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:05.642776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87077187-d3cd-483f-af82-eaea36426618 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:05.643054 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87077187-d3cd-483f-af82-eaea36426618 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:05.643888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87077187-d3cd-483f-af82-eaea36426618 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:05.645810 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87077187-d3cd-483f-af82-eaea36426618 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:05.661029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 771/1539] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:05 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:05.665158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24087514-ca98-4e05-8cc6-fcf71badbe23 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:05.676809 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24087514-ca98-4e05-8cc6-fcf71badbe23 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:05.681460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f231f44e-6798-4fb9-a614-8fd80d6eaf1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:05.682338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f231f44e-6798-4fb9-a614-8fd80d6eaf1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:05.682717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f231f44e-6798-4fb9-a614-8fd80d6eaf1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:05.683078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f231f44e-6798-4fb9-a614-8fd80d6eaf1b admin admin] PATH_INFO: `/v3/projects/32d29f342a2045e484588bf479c5110f/users/2c1298bb18564ac2992a54e1415d5fe0/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:05.700017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f231f44e-6798-4fb9-a614-8fd80d6eaf1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=32d29f342a2045e484588bf479c5110f, user_id=2c1298bb18564ac2992a54e1415d5fe0, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:05.703347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f231f44e-6798-4fb9-a614-8fd80d6eaf1b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:05.725160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 769/1540] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:05 2022] PUT /identity/v3/projects/32d29f342a2045e484588bf479c5110f/users/2c1298bb18564ac2992a54e1415d5fe0/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:05.736206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810daaa-7c90-45bc-8e44-afc2a040f397 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:05.736871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810daaa-7c90-45bc-8e44-afc2a040f397 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:05.737171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810daaa-7c90-45bc-8e44-afc2a040f397 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:05.768686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e810daaa-7c90-45bc-8e44-afc2a040f397 None None] MFA Rules not processed for user `2c1298bb18564ac2992a54e1415d5fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:05.770078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e810daaa-7c90-45bc-8e44-afc2a040f397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:05.896989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 772/1541] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:05.900613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f231f44e-6798-4fb9-a614-8fd80d6eaf1b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:05.992566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810daaa-7c90-45bc-8e44-afc2a040f397 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:05.994345 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e810daaa-7c90-45bc-8e44-afc2a040f397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:05.997494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f231f44e-6798-4fb9-a614-8fd80d6eaf1b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.005238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e40c57-7957-444a-ae5d-d692ee52f732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.005876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e40c57-7957-444a-ae5d-d692ee52f732 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.006203 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e40c57-7957-444a-ae5d-d692ee52f732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.006477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e40c57-7957-444a-ae5d-d692ee52f732 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.007154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e40c57-7957-444a-ae5d-d692ee52f732 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.009309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e40c57-7957-444a-ae5d-d692ee52f732 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.036108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810daaa-7c90-45bc-8e44-afc2a040f397 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.044505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.045623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.046908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.047399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] PATH_INFO: `/v3/projects/cf905b2711344d459391f06f95b5af8f` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.049066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf905b2711344d459391f06f95b5af8f)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.051976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.063325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 770/1542] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:05 2022] POST /identity/v3/projects => generated 385 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:06.066118 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e40c57-7957-444a-ae5d-d692ee52f732 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.127312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf905b2711344d459391f06f95b5af8f', 'request_id': 'req-745f5caa-5e14-4f3e-aa73-426a4acaa032'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:06.137480 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf905b2711344d459391f06f95b5af8f', 'request_id': 'req-745f5caa-5e14-4f3e-aa73-426a4acaa032'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:06.149206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf905b2711344d459391f06f95b5af8f', 'request_id': 'req-745f5caa-5e14-4f3e-aa73-426a4acaa032'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:06.177369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] The token cache is being invalidate because project cf905b2711344d459391f06f95b5af8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:06.178847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:06.180202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-745f5caa-5e14-4f3e-aa73-426a4acaa032 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:06.187751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 773/1543] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:05 2022] DELETE /identity/v3/projects/cf905b2711344d459391f06f95b5af8f => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:06.189792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e44a9f-cbc4-48f9-ae50-d99806e192c5 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.190311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e44a9f-cbc4-48f9-ae50-d99806e192c5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.190812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e44a9f-cbc4-48f9-ae50-d99806e192c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.214335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-65e44a9f-cbc4-48f9-ae50-d99806e192c5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:06.215748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-65e44a9f-cbc4-48f9-ae50-d99806e192c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:06.219861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e40c57-7957-444a-ae5d-d692ee52f732 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.284680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:06.315735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca77b8d3-e996-4cf1-833e-1280d234c3c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.316409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca77b8d3-e996-4cf1-833e-1280d234c3c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.316805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca77b8d3-e996-4cf1-833e-1280d234c3c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.317151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca77b8d3-e996-4cf1-833e-1280d234c3c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.317901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca77b8d3-e996-4cf1-833e-1280d234c3c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.320444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca77b8d3-e996-4cf1-833e-1280d234c3c7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.325919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 774/1544] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:06.327820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc73a3d9-dd2f-4095-a635-19776e763875 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.328307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc73a3d9-dd2f-4095-a635-19776e763875 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.328772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc73a3d9-dd2f-4095-a635-19776e763875 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.333130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:06.350884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 771/1545] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:06 2022] POST /identity/v3/users => generated 299 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:06.354156 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95b2878-8203-4ad8-a76b-b6478255f396 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.354756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95b2878-8203-4ad8-a76b-b6478255f396 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.355183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95b2878-8203-4ad8-a76b-b6478255f396 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.385382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-cc73a3d9-dd2f-4095-a635-19776e763875 None None] MFA Rules not processed for user `0e0407fdc07f4ad98d09fdabbc9952fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:06.389537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cc73a3d9-dd2f-4095-a635-19776e763875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:06.398450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-b95b2878-8203-4ad8-a76b-b6478255f396 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:06.400607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b95b2878-8203-4ad8-a76b-b6478255f396 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:06.412017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 772/1546] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:06.413309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95b2878-8203-4ad8-a76b-b6478255f396 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.414974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b95b2878-8203-4ad8-a76b-b6478255f396 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:06.430957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95b2878-8203-4ad8-a76b-b6478255f396 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.439162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdac2b25-b5d5-4a33-be94-84bccc7bf51a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.440243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac2b25-b5d5-4a33-be94-84bccc7bf51a admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.440732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac2b25-b5d5-4a33-be94-84bccc7bf51a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.441194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac2b25-b5d5-4a33-be94-84bccc7bf51a admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.442256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdac2b25-b5d5-4a33-be94-84bccc7bf51a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.444931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdac2b25-b5d5-4a33-be94-84bccc7bf51a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.465131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 773/1547] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:06 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:06.469097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdac2b25-b5d5-4a33-be94-84bccc7bf51a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.498705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdac2b25-b5d5-4a33-be94-84bccc7bf51a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.502564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 775/1548] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:06.503606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc73a3d9-dd2f-4095-a635-19776e763875 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.518091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.520009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.520009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.520009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.520009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:06.520323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.521841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.536650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc73a3d9-dd2f-4095-a635-19776e763875 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.559567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 774/1549] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:06.560567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.565079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:06.566972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be549111-8e11-4be0-8a64-412cba0fd752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.567832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be549111-8e11-4be0-8a64-412cba0fd752 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.568499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be549111-8e11-4be0-8a64-412cba0fd752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.568978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be549111-8e11-4be0-8a64-412cba0fd752 admin admin] PATH_INFO: `/v3/projects/c00613023cdb49eb9aa91dae93318a5d/users/b356a314bb5742e69187df84baa0588d/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.591025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be549111-8e11-4be0-8a64-412cba0fd752 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c00613023cdb49eb9aa91dae93318a5d, user_id=b356a314bb5742e69187df84baa0588d, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.594525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be549111-8e11-4be0-8a64-412cba0fd752 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.604202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d566f-7900-40e5-b3bb-e32a97c8362b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.618651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a118342-8a8d-41aa-99dc-b31cb879432c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.620851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a118342-8a8d-41aa-99dc-b31cb879432c admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.622706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a118342-8a8d-41aa-99dc-b31cb879432c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.623031 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a118342-8a8d-41aa-99dc-b31cb879432c admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.627098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a118342-8a8d-41aa-99dc-b31cb879432c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.629236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 776/1550] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:06 2022] PUT /identity/v3/projects/c00613023cdb49eb9aa91dae93318a5d/users/b356a314bb5742e69187df84baa0588d/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:06.629875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be549111-8e11-4be0-8a64-412cba0fd752 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.632113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a118342-8a8d-41aa-99dc-b31cb879432c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.688115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 775/1551] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:06 2022] POST /identity/v3/projects => generated 373 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:06.690267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a118342-8a8d-41aa-99dc-b31cb879432c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.711592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be549111-8e11-4be0-8a64-412cba0fd752 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.771795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a118342-8a8d-41aa-99dc-b31cb879432c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.777806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62677e78-22e1-417e-ae6c-d9207276b79c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.778664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62677e78-22e1-417e-ae6c-d9207276b79c admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.779225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62677e78-22e1-417e-ae6c-d9207276b79c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.779225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62677e78-22e1-417e-ae6c-d9207276b79c admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.780186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62677e78-22e1-417e-ae6c-d9207276b79c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.782615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62677e78-22e1-417e-ae6c-d9207276b79c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.799700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 776/1552] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:06 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:06.800681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62677e78-22e1-417e-ae6c-d9207276b79c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.803347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:06.812173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690afd19-01b7-42e0-bf47-58ea0fc45523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.813242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690afd19-01b7-42e0-bf47-58ea0fc45523 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.813649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690afd19-01b7-42e0-bf47-58ea0fc45523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.814806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690afd19-01b7-42e0-bf47-58ea0fc45523 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.815937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690afd19-01b7-42e0-bf47-58ea0fc45523 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.817204 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62677e78-22e1-417e-ae6c-d9207276b79c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.819045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690afd19-01b7-42e0-bf47-58ea0fc45523 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.823412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f702068d-85d9-446c-b7c2-9640d8e49847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.824263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f702068d-85d9-446c-b7c2-9640d8e49847 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.824687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f702068d-85d9-446c-b7c2-9640d8e49847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.825212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f702068d-85d9-446c-b7c2-9640d8e49847 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.826006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f702068d-85d9-446c-b7c2-9640d8e49847 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.828506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f702068d-85d9-446c-b7c2-9640d8e49847 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.857092 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 777/1553] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:06 2022] POST /identity/v3/users => generated 293 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:06.861273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f702068d-85d9-446c-b7c2-9640d8e49847 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.870316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 777/1554] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:06 2022] POST /identity/v3/projects => generated 385 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:06.871355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690afd19-01b7-42e0-bf47-58ea0fc45523 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:06.873460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f702068d-85d9-446c-b7c2-9640d8e49847 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.954416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34794939-8572-4c40-8549-a8f61eb15155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:06.955734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34794939-8572-4c40-8549-a8f61eb15155 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:06.956026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34794939-8572-4c40-8549-a8f61eb15155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:06.956467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34794939-8572-4c40-8549-a8f61eb15155 admin admin] PATH_INFO: `/v3/projects/c00613023cdb49eb9aa91dae93318a5d/users/b356a314bb5742e69187df84baa0588d/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:06.958535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34794939-8572-4c40-8549-a8f61eb15155 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c00613023cdb49eb9aa91dae93318a5d, user_id=b356a314bb5742e69187df84baa0588d, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:06.961792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34794939-8572-4c40-8549-a8f61eb15155 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:06.980716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690afd19-01b7-42e0-bf47-58ea0fc45523 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:06.991824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 778/1555] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:06 2022] PUT /identity/v3/projects/c00613023cdb49eb9aa91dae93318a5d/users/b356a314bb5742e69187df84baa0588d/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:06.991824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34794939-8572-4c40-8549-a8f61eb15155 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.031600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:07.083449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5978d78-9834-4b3c-9ca3-7cbc1fdfa2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.084456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5978d78-9834-4b3c-9ca3-7cbc1fdfa2b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.084877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5978d78-9834-4b3c-9ca3-7cbc1fdfa2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.085533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5978d78-9834-4b3c-9ca3-7cbc1fdfa2b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.086172 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5978d78-9834-4b3c-9ca3-7cbc1fdfa2b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.089293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5978d78-9834-4b3c-9ca3-7cbc1fdfa2b8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.103818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 778/1556] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:06 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:07.104991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0609ceb2-113f-4373-975b-6c7fa5844c93 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.105435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0609ceb2-113f-4373-975b-6c7fa5844c93 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.105601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0609ceb2-113f-4373-975b-6c7fa5844c93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.114236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34794939-8572-4c40-8549-a8f61eb15155 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.122909 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59b080b-af19-4b32-bcca-14c7049260c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.123868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59b080b-af19-4b32-bcca-14c7049260c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.124243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59b080b-af19-4b32-bcca-14c7049260c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.124692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59b080b-af19-4b32-bcca-14c7049260c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.125618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59b080b-af19-4b32-bcca-14c7049260c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.128803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59b080b-af19-4b32-bcca-14c7049260c4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.158910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0609ceb2-113f-4373-975b-6c7fa5844c93 None None] MFA Rules not processed for user `b356a314bb5742e69187df84baa0588d`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:07.164963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 779/1557] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:06 2022] POST /identity/v3/users => generated 293 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:07.164963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59b080b-af19-4b32-bcca-14c7049260c4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.165185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0609ceb2-113f-4373-975b-6c7fa5844c93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:07.192326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59b080b-af19-4b32-bcca-14c7049260c4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.197968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f3b3fd-db33-4371-ac11-d593aba441f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.198693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3b3fd-db33-4371-ac11-d593aba441f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.198974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3b3fd-db33-4371-ac11-d593aba441f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.199244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3b3fd-db33-4371-ac11-d593aba441f4 admin admin] PATH_INFO: `/v3/projects/178860dc773c4926b01b6a8e639662ea/users/c837648810924b9ea593e43bc8cd5ba8/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.222215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f3b3fd-db33-4371-ac11-d593aba441f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=178860dc773c4926b01b6a8e639662ea, user_id=c837648810924b9ea593e43bc8cd5ba8, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.225075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f3b3fd-db33-4371-ac11-d593aba441f4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.246765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 780/1558] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:07 2022] PUT /identity/v3/projects/178860dc773c4926b01b6a8e639662ea/users/c837648810924b9ea593e43bc8cd5ba8/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:07.247751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f3b3fd-db33-4371-ac11-d593aba441f4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.287273 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 779/1559] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:07 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:07.300348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0609ceb2-113f-4373-975b-6c7fa5844c93 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.315033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0609ceb2-113f-4373-975b-6c7fa5844c93 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.324274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:07.352161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f3b3fd-db33-4371-ac11-d593aba441f4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.357525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.358274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.358578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.358850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.359494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.361334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.373601 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 781/1560] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:07 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:07.374657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.381419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:07.415171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2455b28e-8c7c-4dcf-b15b-f682080b1842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.415879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2455b28e-8c7c-4dcf-b15b-f682080b1842 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.416228 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2455b28e-8c7c-4dcf-b15b-f682080b1842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.416569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2455b28e-8c7c-4dcf-b15b-f682080b1842 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.417319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2455b28e-8c7c-4dcf-b15b-f682080b1842 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.419051 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2455b28e-8c7c-4dcf-b15b-f682080b1842 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.430330 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 780/1561] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:07 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:07.431362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2455b28e-8c7c-4dcf-b15b-f682080b1842 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.446888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2455b28e-8c7c-4dcf-b15b-f682080b1842 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.453107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71f8372-b7bd-4339-934b-2e004c1280b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.453726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71f8372-b7bd-4339-934b-2e004c1280b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.454011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71f8372-b7bd-4339-934b-2e004c1280b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.454287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71f8372-b7bd-4339-934b-2e004c1280b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.455009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71f8372-b7bd-4339-934b-2e004c1280b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.455755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233f651f-268a-437c-ae91-e5e7c1d15088 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.456637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71f8372-b7bd-4339-934b-2e004c1280b4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.460075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc7ff5a-b377-4f46-be63-b577fdfd394d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.460422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7ff5a-b377-4f46-be63-b577fdfd394d service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.460822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7ff5a-b377-4f46-be63-b577fdfd394d service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.461095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7ff5a-b377-4f46-be63-b577fdfd394d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.490516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 781/1562] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:07 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:07.490516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71f8372-b7bd-4339-934b-2e004c1280b4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.497037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71f8372-b7bd-4339-934b-2e004c1280b4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.501413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:07.504189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3992b1b0-ec3c-49f5-8def-02cafcb3772c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.504984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3992b1b0-ec3c-49f5-8def-02cafcb3772c admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.505228 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3992b1b0-ec3c-49f5-8def-02cafcb3772c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.505526 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3992b1b0-ec3c-49f5-8def-02cafcb3772c admin admin] PATH_INFO: `/v3/projects/178860dc773c4926b01b6a8e639662ea/users/c837648810924b9ea593e43bc8cd5ba8/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.507941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3992b1b0-ec3c-49f5-8def-02cafcb3772c admin admin] RBAC: Authorizing `identity:create_grant(project_id=178860dc773c4926b01b6a8e639662ea, user_id=c837648810924b9ea593e43bc8cd5ba8, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.510921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3992b1b0-ec3c-49f5-8def-02cafcb3772c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.527376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 782/1563] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:07 2022] PUT /identity/v3/projects/178860dc773c4926b01b6a8e639662ea/users/c837648810924b9ea593e43bc8cd5ba8/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:07.527376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3992b1b0-ec3c-49f5-8def-02cafcb3772c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.553205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc7ff5a-b377-4f46-be63-b577fdfd394d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.555188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc7ff5a-b377-4f46-be63-b577fdfd394d service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.569881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 782/1564] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:07 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:07.632902 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3992b1b0-ec3c-49f5-8def-02cafcb3772c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.643692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71915a5d-d8fb-4087-a17e-b9a5c90d60a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.644407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71915a5d-d8fb-4087-a17e-b9a5c90d60a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.644690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71915a5d-d8fb-4087-a17e-b9a5c90d60a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.644964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71915a5d-d8fb-4087-a17e-b9a5c90d60a9 admin admin] PATH_INFO: `/v3/projects/740d572054e54deaa181f7059e93b4d2/users/6e63d3e6586f40d283e4ab7020b517e0/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.664476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71915a5d-d8fb-4087-a17e-b9a5c90d60a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=740d572054e54deaa181f7059e93b4d2, user_id=6e63d3e6586f40d283e4ab7020b517e0, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.667820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71915a5d-d8fb-4087-a17e-b9a5c90d60a9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:07.688262 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 783/1565] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:07 2022] PUT /identity/v3/projects/740d572054e54deaa181f7059e93b4d2/users/6e63d3e6586f40d283e4ab7020b517e0/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:07.699757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec75416-815d-4921-be92-4469b47fd0c9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.700330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec75416-815d-4921-be92-4469b47fd0c9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.700859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec75416-815d-4921-be92-4469b47fd0c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.754418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-0ec75416-815d-4921-be92-4469b47fd0c9 None None] MFA Rules not processed for user `6e63d3e6586f40d283e4ab7020b517e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:07.755880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0ec75416-815d-4921-be92-4469b47fd0c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:07.854073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 783/1566] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:07.857855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71915a5d-d8fb-4087-a17e-b9a5c90d60a9 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:07.982961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71915a5d-d8fb-4087-a17e-b9a5c90d60a9 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:07.991913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbb9f41-f76f-4c41-afe7-c74fbecd5da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:07.993611 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb9f41-f76f-4c41-afe7-c74fbecd5da0 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:07.994161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb9f41-f76f-4c41-afe7-c74fbecd5da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:07.994688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbb9f41-f76f-4c41-afe7-c74fbecd5da0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:07.996662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbb9f41-f76f-4c41-afe7-c74fbecd5da0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:07.999774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbb9f41-f76f-4c41-afe7-c74fbecd5da0 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:08.015324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:08.062627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 784/1567] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:07 2022] POST /identity/v3/projects => generated 385 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:08.066615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec75416-815d-4921-be92-4469b47fd0c9 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:08.089370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec75416-815d-4921-be92-4469b47fd0c9 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:08.173822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c0df6-5d0c-4338-90cd-42799a096e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:08.175309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c0df6-5d0c-4338-90cd-42799a096e6a admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:08.179229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c0df6-5d0c-4338-90cd-42799a096e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:08.179789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c0df6-5d0c-4338-90cd-42799a096e6a admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:08.181305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681c0df6-5d0c-4338-90cd-42799a096e6a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:08.183521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681c0df6-5d0c-4338-90cd-42799a096e6a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:08.222585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 784/1568] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:08 2022] POST /identity/v3/users => generated 299 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:08.223208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbb9f41-f76f-4c41-afe7-c74fbecd5da0 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:08.270369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbb9f41-f76f-4c41-afe7-c74fbecd5da0 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:08.295687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38278e-47ee-40d3-9fb0-80ee3236f8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:08.298019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38278e-47ee-40d3-9fb0-80ee3236f8f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:08.298372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38278e-47ee-40d3-9fb0-80ee3236f8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:08.299376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38278e-47ee-40d3-9fb0-80ee3236f8f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:08.302269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff38278e-47ee-40d3-9fb0-80ee3236f8f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:08.309387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff38278e-47ee-40d3-9fb0-80ee3236f8f7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:08.342606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 785/1569] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:08 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:08.349217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c0df6-5d0c-4338-90cd-42799a096e6a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:08.371430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c0df6-5d0c-4338-90cd-42799a096e6a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:08.395617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d45ea6-664c-47e5-97af-59ec6c4bf811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:08.396303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d45ea6-664c-47e5-97af-59ec6c4bf811 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:08.396584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d45ea6-664c-47e5-97af-59ec6c4bf811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:08.396861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d45ea6-664c-47e5-97af-59ec6c4bf811 admin admin] PATH_INFO: `/v3/projects/a7b6645b1cd0413e97d99f10e5e9cc86/users/fbd494bba459433a8be305b120a6df67/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:08.441226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d45ea6-664c-47e5-97af-59ec6c4bf811 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7b6645b1cd0413e97d99f10e5e9cc86, user_id=fbd494bba459433a8be305b120a6df67, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:08.456823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d45ea6-664c-47e5-97af-59ec6c4bf811 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:08.485328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 785/1570] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:08 2022] PUT /identity/v3/projects/a7b6645b1cd0413e97d99f10e5e9cc86/users/fbd494bba459433a8be305b120a6df67/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:08.489886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38278e-47ee-40d3-9fb0-80ee3236f8f7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:08.507436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38278e-47ee-40d3-9fb0-80ee3236f8f7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:08.607119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77dfd17-a73d-45c6-9aaf-047ba658b4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:08.607759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77dfd17-a73d-45c6-9aaf-047ba658b4ce admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:08.609520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77dfd17-a73d-45c6-9aaf-047ba658b4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:08.609520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77dfd17-a73d-45c6-9aaf-047ba658b4ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:08.610119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77dfd17-a73d-45c6-9aaf-047ba658b4ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:08.614399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77dfd17-a73d-45c6-9aaf-047ba658b4ce admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:08.629069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 786/1571] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:08 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:08.632840 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d45ea6-664c-47e5-97af-59ec6c4bf811 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:08.652979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d45ea6-664c-47e5-97af-59ec6c4bf811 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:08.661577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4729d8b-1afd-4da2-831e-8c1e65b8d9c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:08.662284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4729d8b-1afd-4da2-831e-8c1e65b8d9c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:08.662587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4729d8b-1afd-4da2-831e-8c1e65b8d9c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:08.663312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4729d8b-1afd-4da2-831e-8c1e65b8d9c2 admin admin] PATH_INFO: `/v3/projects/a7b6645b1cd0413e97d99f10e5e9cc86/users/fbd494bba459433a8be305b120a6df67/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:08.670701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4729d8b-1afd-4da2-831e-8c1e65b8d9c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7b6645b1cd0413e97d99f10e5e9cc86, user_id=fbd494bba459433a8be305b120a6df67, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:08.675281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4729d8b-1afd-4da2-831e-8c1e65b8d9c2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:08.680042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77dfd17-a73d-45c6-9aaf-047ba658b4ce admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:08.698540 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 786/1572] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:08 2022] PUT /identity/v3/projects/a7b6645b1cd0413e97d99f10e5e9cc86/users/fbd494bba459433a8be305b120a6df67/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:08.704041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d2825-4982-4cd5-af52-1dacb90138d8 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:08.705755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d2825-4982-4cd5-af52-1dacb90138d8 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:08.705893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77dfd17-a73d-45c6-9aaf-047ba658b4ce admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:08.705988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d2825-4982-4cd5-af52-1dacb90138d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:08.727123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:08.748683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-9e8d2825-4982-4cd5-af52-1dacb90138d8 None None] MFA Rules not processed for user `fbd494bba459433a8be305b120a6df67`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:08.750033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9e8d2825-4982-4cd5-af52-1dacb90138d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:08.800475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:08.801398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:08.801809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:08.802163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] PATH_INFO: `/v3/users/2c1298bb18564ac2992a54e1415d5fe0` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:08.804083 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c1298bb18564ac2992a54e1415d5fe0)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:08.806307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:08.870011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 787/1573] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:08 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:08.885804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8d2825-4982-4cd5-af52-1dacb90138d8 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:08.932362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c1298bb18564ac2992a54e1415d5fe0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:08.949435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c1298bb18564ac2992a54e1415d5fe0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:08.954939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c1298bb18564ac2992a54e1415d5fe0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:08.955826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c1298bb18564ac2992a54e1415d5fe0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:08.982517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e3092e22-b95e-4088-9724-0ad27f1e6b8d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c1298bb18564ac2992a54e1415d5fe0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:08.997627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:09.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8d2825-4982-4cd5-af52-1dacb90138d8 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:09.006650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:09.007619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:09.007954 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:09.008238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:09.010625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:09.013455 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 787/1574] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:08 2022] DELETE /identity/v3/users/2c1298bb18564ac2992a54e1415d5fe0 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:09.017588 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb18d6-785c-4b0c-9f9d-8c5fde383bf7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:09.017874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb18d6-785c-4b0c-9f9d-8c5fde383bf7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:09.018146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb18d6-785c-4b0c-9f9d-8c5fde383bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:09.030541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:09.032219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:09.042631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5dbb18d6-785c-4b0c-9f9d-8c5fde383bf7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:09.043863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5dbb18d6-785c-4b0c-9f9d-8c5fde383bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:09.047954 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 788/1575] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:08 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:09.084071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:09.086533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:09.108150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e91bcd8-561b-4a4b-872b-ef246957807d service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:09.149186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 788/1576] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:09.164925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbb18d6-785c-4b0c-9f9d-8c5fde383bf7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:09.167568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5dbb18d6-785c-4b0c-9f9d-8c5fde383bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:09.206839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:09.208577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:09.209017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:09.209415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] PATH_INFO: `/v3/users/6e63d3e6586f40d283e4ab7020b517e0` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:09.211146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e63d3e6586f40d283e4ab7020b517e0)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:09.214173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:09.248806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbb18d6-785c-4b0c-9f9d-8c5fde383bf7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:09.255281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1761d97b-003c-401d-8e2e-99efec5e2ed8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:09.256046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1761d97b-003c-401d-8e2e-99efec5e2ed8 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:09.256351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1761d97b-003c-401d-8e2e-99efec5e2ed8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:09.256642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1761d97b-003c-401d-8e2e-99efec5e2ed8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:09.272293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1761d97b-003c-401d-8e2e-99efec5e2ed8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:09.275197 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1761d97b-003c-401d-8e2e-99efec5e2ed8 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:09.282226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:09.288316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 789/1577] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:09.298894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e63d3e6586f40d283e4ab7020b517e0'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:09.311617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e63d3e6586f40d283e4ab7020b517e0'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:09.313343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e63d3e6586f40d283e4ab7020b517e0'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:09.316326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e63d3e6586f40d283e4ab7020b517e0'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:09.329144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2a1e502f-ce82-4799-8e4d-9973bdd9191e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e63d3e6586f40d283e4ab7020b517e0'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:09.355975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 789/1578] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:09 2022] DELETE /identity/v3/users/6e63d3e6586f40d283e4ab7020b517e0 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:09.357762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178737ab-c3df-440c-b1f9-dff80e990dbf None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:09.358053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178737ab-c3df-440c-b1f9-dff80e990dbf None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:09.358338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178737ab-c3df-440c-b1f9-dff80e990dbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:09.380661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-178737ab-c3df-440c-b1f9-dff80e990dbf None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:09.382707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-178737ab-c3df-440c-b1f9-dff80e990dbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:09.428206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af15353-07e0-40a9-b418-459e38c230f1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:09.429477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af15353-07e0-40a9-b418-459e38c230f1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:09.429970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af15353-07e0-40a9-b418-459e38c230f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:09.483266 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7af15353-07e0-40a9-b418-459e38c230f1 None None] MFA Rules not processed for user `c837648810924b9ea593e43bc8cd5ba8`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:09.484478 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7af15353-07e0-40a9-b418-459e38c230f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:09.514022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 790/1579] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:09.566819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178737ab-c3df-440c-b1f9-dff80e990dbf None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:09.568496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-178737ab-c3df-440c-b1f9-dff80e990dbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:09.631953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 790/1580] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:09 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:09.644421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af15353-07e0-40a9-b418-459e38c230f1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:09.646592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7af15353-07e0-40a9-b418-459e38c230f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:09.681457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:09.740095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178737ab-c3df-440c-b1f9-dff80e990dbf None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:09.755688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:09.757901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:09.758314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:09.759809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:09.791242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:09.795178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:09.825815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 791/1581] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:09.831361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af15353-07e0-40a9-b418-459e38c230f1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:09.840512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:09.840512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:09.840512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:09.842121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:09.844227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:10.004176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:10.038082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:10.039157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:10.066645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 791/1582] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:09 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:10.541805 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:10.547327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:10.579356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff112d5-1f4e-425e-b24a-f85a4eb3a633 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:10.584667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:10.585323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:10.585606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:10.585884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] PATH_INFO: `/v3/projects/32d29f342a2045e484588bf479c5110f` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:10.587176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32d29f342a2045e484588bf479c5110f)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:10.589258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:10.655420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:10.657131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:10.676065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e345c745-4b56-43f5-9bd2-d58c31dfd885 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:10.690038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32d29f342a2045e484588bf479c5110f', 'request_id': 'req-07f5f95c-b09e-4288-8130-7c56ce697940'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.698198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32d29f342a2045e484588bf479c5110f', 'request_id': 'req-07f5f95c-b09e-4288-8130-7c56ce697940'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.712716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32d29f342a2045e484588bf479c5110f', 'request_id': 'req-07f5f95c-b09e-4288-8130-7c56ce697940'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.731671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] The token cache is being invalidate because project 32d29f342a2045e484588bf479c5110f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:10.732094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.733176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-07f5f95c-b09e-4288-8130-7c56ce697940 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.735304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 792/1583] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:10 2022] DELETE /identity/v3/projects/32d29f342a2045e484588bf479c5110f => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:10.739314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adf96e-ff9d-43c2-85ed-ec41b6f3ac26 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:10.740030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adf96e-ff9d-43c2-85ed-ec41b6f3ac26 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:10.740654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adf96e-ff9d-43c2-85ed-ec41b6f3ac26 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:10.743196 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 793/1584] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:41:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:41:10.745701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adf96e-ff9d-43c2-85ed-ec41b6f3ac26 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:10.747959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-50adf96e-ff9d-43c2-85ed-ec41b6f3ac26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:10.765175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:10.766038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:10.766686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:10.767144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] PATH_INFO: `/v3/projects/740d572054e54deaa181f7059e93b4d2` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:10.769209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] RBAC: Authorizing `identity:delete_project(project_id=740d572054e54deaa181f7059e93b4d2)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:10.771700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:10.842800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adf96e-ff9d-43c2-85ed-ec41b6f3ac26 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:10.850354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '740d572054e54deaa181f7059e93b4d2', 'request_id': 'req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.857587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '740d572054e54deaa181f7059e93b4d2', 'request_id': 'req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.864780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '740d572054e54deaa181f7059e93b4d2', 'request_id': 'req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.879925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] The token cache is being invalidate because project 740d572054e54deaa181f7059e93b4d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:10.880766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.881530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9dcf0b5b-4229-4fba-a672-4ceb8c486b0e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:10.883153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 792/1585] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:10 2022] DELETE /identity/v3/projects/740d572054e54deaa181f7059e93b4d2 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:10.886291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473b7ab3-ee5d-4ed0-b6f9-e16e0f6da83e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:10.886291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473b7ab3-ee5d-4ed0-b6f9-e16e0f6da83e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:10.886291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473b7ab3-ee5d-4ed0-b6f9-e16e0f6da83e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:10.911192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:10.911400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:10.911640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:10.911820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] PATH_INFO: `/v3/users/b356a314bb5742e69187df84baa0588d` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:10.913010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b356a314bb5742e69187df84baa0588d)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:10.915184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:10.918137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-473b7ab3-ee5d-4ed0-b6f9-e16e0f6da83e None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:10.919401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-473b7ab3-ee5d-4ed0-b6f9-e16e0f6da83e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:10.933660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:10.986755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b356a314bb5742e69187df84baa0588d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.011520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b356a314bb5742e69187df84baa0588d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.012683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b356a314bb5742e69187df84baa0588d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.013650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b356a314bb5742e69187df84baa0588d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.020850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 793/1586] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:41:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:11.022512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473b7ab3-ee5d-4ed0-b6f9-e16e0f6da83e None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:11.023959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-473b7ab3-ee5d-4ed0-b6f9-e16e0f6da83e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:11.033863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b356a314bb5742e69187df84baa0588d'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.040892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473b7ab3-ee5d-4ed0-b6f9-e16e0f6da83e None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:11.047462 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:11.048662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:11.049097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:11.049986 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] PATH_INFO: `/v3/users/fbd494bba459433a8be305b120a6df67` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:11.052175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbd494bba459433a8be305b120a6df67)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:11.055264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:11.070411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 794/1587] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:10 2022] DELETE /identity/v3/users/b356a314bb5742e69187df84baa0588d => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:11.076661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:11.076812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:11.083375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:11.151608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbd494bba459433a8be305b120a6df67'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.169453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbd494bba459433a8be305b120a6df67'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.170559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbd494bba459433a8be305b120a6df67'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.171317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbd494bba459433a8be305b120a6df67'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.187440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbd494bba459433a8be305b120a6df67'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.260305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 794/1588] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:11 2022] DELETE /identity/v3/users/fbd494bba459433a8be305b120a6df67 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:11.320592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce66c19b-47c9-488f-87e9-473742e78479 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:11.512648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:11.513302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:11.513606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:11.513879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:11.516707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:11.522848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:11.523319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:11.537435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:11.539236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:11.584473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 795/1589] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:11.694282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df7d5bb-d5ef-448e-b33e-2e9acf59b721 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:11.703707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:11.704489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:11.704877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:11.705196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] PATH_INFO: `/v3/projects/c00613023cdb49eb9aa91dae93318a5d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:11.706527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=c00613023cdb49eb9aa91dae93318a5d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:11.708436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:11.788335 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c00613023cdb49eb9aa91dae93318a5d', 'request_id': 'req-2b760592-820c-4f93-82fb-b3e4bc3529ed'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.800004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:11.801488 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c00613023cdb49eb9aa91dae93318a5d', 'request_id': 'req-2b760592-820c-4f93-82fb-b3e4bc3529ed'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.801605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:11.819351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c00613023cdb49eb9aa91dae93318a5d', 'request_id': 'req-2b760592-820c-4f93-82fb-b3e4bc3529ed'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.831026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:11.877302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] The token cache is being invalidate because project c00613023cdb49eb9aa91dae93318a5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:11.877645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.878515 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:11.886465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 795/1590] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:11 2022] DELETE /identity/v3/projects/c00613023cdb49eb9aa91dae93318a5d => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:11.889134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:11.893961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:11.925231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:11.942137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654e98d7-b066-4044-b2a0-c785add45ba8 service nova] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:11.988813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b760592-820c-4f93-82fb-b3e4bc3529ed admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:11.999259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:12.000894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.002050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.002883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] PATH_INFO: `/v3/projects/a7b6645b1cd0413e97d99f10e5e9cc86` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.004935 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7b6645b1cd0413e97d99f10e5e9cc86)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:12.007560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:12.020569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e03d23-c047-469e-b985-dceb6a74591d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:12.021199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e03d23-c047-469e-b985-dceb6a74591d service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.021867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e03d23-c047-469e-b985-dceb6a74591d service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.022180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e03d23-c047-469e-b985-dceb6a74591d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.024782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-50e03d23-c047-469e-b985-dceb6a74591d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:12.082375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7b6645b1cd0413e97d99f10e5e9cc86', 'request_id': 'req-082abef5-1dbb-4511-bd93-5d9d05b52a4b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:12.102590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7b6645b1cd0413e97d99f10e5e9cc86', 'request_id': 'req-082abef5-1dbb-4511-bd93-5d9d05b52a4b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:12.119034 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7b6645b1cd0413e97d99f10e5e9cc86', 'request_id': 'req-082abef5-1dbb-4511-bd93-5d9d05b52a4b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:12.146016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] The token cache is being invalidate because project a7b6645b1cd0413e97d99f10e5e9cc86 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:12.146359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:12.147168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-082abef5-1dbb-4511-bd93-5d9d05b52a4b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:12.150650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 796/1591] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:11 2022] DELETE /identity/v3/projects/a7b6645b1cd0413e97d99f10e5e9cc86 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:12.152946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7957a-34f5-4429-9fe0-611e34397eca None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.153181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7957a-34f5-4429-9fe0-611e34397eca None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.154285 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a7957a-34f5-4429-9fe0-611e34397eca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.169838 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e03d23-c047-469e-b985-dceb6a74591d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:12.172815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e03d23-c047-469e-b985-dceb6a74591d service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:12.200681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 796/1592] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:11 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:12.202074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6585a75-d4fc-4c5a-9e6d-44170c6c4949 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.202360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6585a75-d4fc-4c5a-9e6d-44170c6c4949 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.202662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6585a75-d4fc-4c5a-9e6d-44170c6c4949 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.203742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-12a7957a-34f5-4429-9fe0-611e34397eca None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:12.206178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-12a7957a-34f5-4429-9fe0-611e34397eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:12.271837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b6585a75-d4fc-4c5a-9e6d-44170c6c4949 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:12.273088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b6585a75-d4fc-4c5a-9e6d-44170c6c4949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:12.364395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 797/1593] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:12.371451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b6da0-d7fc-4e31-a710-ebf437a7584f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.372073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b6da0-d7fc-4e31-a710-ebf437a7584f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.372370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b6da0-d7fc-4e31-a710-ebf437a7584f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.408846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-b81b6da0-d7fc-4e31-a710-ebf437a7584f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:12.410603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b81b6da0-d7fc-4e31-a710-ebf437a7584f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:12.414141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 797/1594] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:12.424056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173e8219-8353-4f10-9f14-861959892b92 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.424564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173e8219-8353-4f10-9f14-861959892b92 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.425028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173e8219-8353-4f10-9f14-861959892b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.438134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 798/1595] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:12.443208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81b6da0-d7fc-4e31-a710-ebf437a7584f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:12.483533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81b6da0-d7fc-4e31-a710-ebf437a7584f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:12.501110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:12.501894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.502253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.503430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.503430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:12.503948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:12.505708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:12.506582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-173e8219-8353-4f10-9f14-861959892b92 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:12.509028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-173e8219-8353-4f10-9f14-861959892b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:12.541763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 799/1596] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:12.541954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 798/1597] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:12.542032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:12.547617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173e8219-8353-4f10-9f14-861959892b92 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:12.561795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6397434f-8d93-451d-b14c-c026fa3f3e38 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:12.574994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:12.575598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.575876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.576214 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.577279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173e8219-8353-4f10-9f14-861959892b92 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:12.579728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:12.580434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:12.582577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:12.590687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58422d0c-2acd-48c7-a62b-315de1fcf54e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:12.591623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58422d0c-2acd-48c7-a62b-315de1fcf54e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.592151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58422d0c-2acd-48c7-a62b-315de1fcf54e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.592624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58422d0c-2acd-48c7-a62b-315de1fcf54e admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.593501 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58422d0c-2acd-48c7-a62b-315de1fcf54e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:12.596527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58422d0c-2acd-48c7-a62b-315de1fcf54e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:12.619631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 800/1598] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:41:12.624835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:12.657246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a33cf86-6c8e-4e3e-b4f7-0dd8e5cbb480 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:12.662641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:12.692621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 799/1599] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:12 2022] POST /identity/v3/projects => generated 385 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:12.695684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58422d0c-2acd-48c7-a62b-315de1fcf54e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:12.697957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eed8aeb-1904-4f9b-b079-b681c8b6a6c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:12.699715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed8aeb-1904-4f9b-b079-b681c8b6a6c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.700212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed8aeb-1904-4f9b-b079-b681c8b6a6c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.700722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed8aeb-1904-4f9b-b079-b681c8b6a6c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.701800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eed8aeb-1904-4f9b-b079-b681c8b6a6c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:12.710287 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eed8aeb-1904-4f9b-b079-b681c8b6a6c3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:12.756146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:12.797300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 801/1600] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:12 2022] POST /identity/v3/projects => generated 375 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:12.802682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eed8aeb-1904-4f9b-b079-b681c8b6a6c3 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:12.845951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58422d0c-2acd-48c7-a62b-315de1fcf54e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:12.943555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f68a66-d62d-440f-ac2a-ff1e803b8add admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:12.944260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f68a66-d62d-440f-ac2a-ff1e803b8add admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.944582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f68a66-d62d-440f-ac2a-ff1e803b8add admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.944880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f68a66-d62d-440f-ac2a-ff1e803b8add admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.945693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f68a66-d62d-440f-ac2a-ff1e803b8add admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:12.948764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f68a66-d62d-440f-ac2a-ff1e803b8add admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:12.979447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eed8aeb-1904-4f9b-b079-b681c8b6a6c3 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:12.986763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728c2611-bac7-4407-b15f-5cd9e413c429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:12.988529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728c2611-bac7-4407-b15f-5cd9e413c429 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:12.988912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728c2611-bac7-4407-b15f-5cd9e413c429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:12.989733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728c2611-bac7-4407-b15f-5cd9e413c429 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:12.990984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728c2611-bac7-4407-b15f-5cd9e413c429 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:12.997718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 800/1601] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:12 2022] POST /identity/v3/users => generated 293 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:12.997872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728c2611-bac7-4407-b15f-5cd9e413c429 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.000543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f68a66-d62d-440f-ac2a-ff1e803b8add admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.015929 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f68a66-d62d-440f-ac2a-ff1e803b8add admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.022593 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c7968b-57f0-44d4-8923-2e10651034dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.023224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c7968b-57f0-44d4-8923-2e10651034dc admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.023509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c7968b-57f0-44d4-8923-2e10651034dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.024331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c7968b-57f0-44d4-8923-2e10651034dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.025122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c7968b-57f0-44d4-8923-2e10651034dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.026862 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c7968b-57f0-44d4-8923-2e10651034dc admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.052772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 801/1602] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:13.055106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c7968b-57f0-44d4-8923-2e10651034dc admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.074965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 802/1603] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:12 2022] POST /identity/v3/users => generated 288 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:13.076653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728c2611-bac7-4407-b15f-5cd9e413c429 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c7968b-57f0-44d4-8923-2e10651034dc admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.100312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe9e4a3-addd-4fd1-8e8e-9e83686fd867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.101545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe9e4a3-addd-4fd1-8e8e-9e83686fd867 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.101545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe9e4a3-addd-4fd1-8e8e-9e83686fd867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.102046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe9e4a3-addd-4fd1-8e8e-9e83686fd867 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.113497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe9e4a3-addd-4fd1-8e8e-9e83686fd867 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.119000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe9e4a3-addd-4fd1-8e8e-9e83686fd867 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.119162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728c2611-bac7-4407-b15f-5cd9e413c429 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.130646 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af89af04-8e62-4b61-ae3a-3ad8d59e80f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.131351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af89af04-8e62-4b61-ae3a-3ad8d59e80f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.131704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af89af04-8e62-4b61-ae3a-3ad8d59e80f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.132571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af89af04-8e62-4b61-ae3a-3ad8d59e80f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.135209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af89af04-8e62-4b61-ae3a-3ad8d59e80f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.136933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af89af04-8e62-4b61-ae3a-3ad8d59e80f5 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.137578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 802/1604] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:13.166087 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe9e4a3-addd-4fd1-8e8e-9e83686fd867 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.174199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 803/1605] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:13.175311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af89af04-8e62-4b61-ae3a-3ad8d59e80f5 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.195101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af89af04-8e62-4b61-ae3a-3ad8d59e80f5 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.201339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe9e4a3-addd-4fd1-8e8e-9e83686fd867 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.204970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759880a8-525a-4813-bd3b-a5444bb219ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.205591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759880a8-525a-4813-bd3b-a5444bb219ed admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.205866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759880a8-525a-4813-bd3b-a5444bb219ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.206188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759880a8-525a-4813-bd3b-a5444bb219ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.207160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759880a8-525a-4813-bd3b-a5444bb219ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.208770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759880a8-525a-4813-bd3b-a5444bb219ed admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.216096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be20689f-eb11-4c1b-97a2-5b65fe2bd699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.217126 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be20689f-eb11-4c1b-97a2-5b65fe2bd699 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.218019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be20689f-eb11-4c1b-97a2-5b65fe2bd699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.218319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be20689f-eb11-4c1b-97a2-5b65fe2bd699 admin admin] PATH_INFO: `/v3/projects/f7ceb8b35c74456f92354d47e1802e54/users/9480d0455a214c55adeeaee9a2eace8f/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.237539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 804/1606] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:13.238468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759880a8-525a-4813-bd3b-a5444bb219ed admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.254286 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759880a8-525a-4813-bd3b-a5444bb219ed admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.257047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be20689f-eb11-4c1b-97a2-5b65fe2bd699 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7ceb8b35c74456f92354d47e1802e54, user_id=9480d0455a214c55adeeaee9a2eace8f, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.260029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be20689f-eb11-4c1b-97a2-5b65fe2bd699 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.269723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed23a9de-f7dd-4c8a-8717-aae632486fe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.270426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed23a9de-f7dd-4c8a-8717-aae632486fe3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.270737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed23a9de-f7dd-4c8a-8717-aae632486fe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.271623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed23a9de-f7dd-4c8a-8717-aae632486fe3 admin admin] PATH_INFO: `/v3/projects/8b6e57e5385c44959ce32df7da918bf8/users/d85356c8535d4065b846272a1b559e43/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.290812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 803/1607] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:13 2022] PUT /identity/v3/projects/f7ceb8b35c74456f92354d47e1802e54/users/9480d0455a214c55adeeaee9a2eace8f/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:13.297157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae5afb-e81a-4ac8-8356-d87c5a54d970 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.297438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae5afb-e81a-4ac8-8356-d87c5a54d970 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.297712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae5afb-e81a-4ac8-8356-d87c5a54d970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.316406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed23a9de-f7dd-4c8a-8717-aae632486fe3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b6e57e5385c44959ce32df7da918bf8, user_id=d85356c8535d4065b846272a1b559e43, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.327472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed23a9de-f7dd-4c8a-8717-aae632486fe3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.362147 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 805/1608] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:13 2022] PUT /identity/v3/projects/8b6e57e5385c44959ce32df7da918bf8/users/d85356c8535d4065b846272a1b559e43/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:13.373618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef63f79-abc2-4b1c-b9fa-f9c4815f9410 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.373905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef63f79-abc2-4b1c-b9fa-f9c4815f9410 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.374184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef63f79-abc2-4b1c-b9fa-f9c4815f9410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.390150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-65ae5afb-e81a-4ac8-8356-d87c5a54d970 None None] MFA Rules not processed for user `9480d0455a214c55adeeaee9a2eace8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:13.391501 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-65ae5afb-e81a-4ac8-8356-d87c5a54d970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:13.425420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-4ef63f79-abc2-4b1c-b9fa-f9c4815f9410 None None] MFA Rules not processed for user `d85356c8535d4065b846272a1b559e43`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:13.426904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4ef63f79-abc2-4b1c-b9fa-f9c4815f9410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:13.474681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:13.515322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 804/1609] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:13 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:13.516264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ae5afb-e81a-4ac8-8356-d87c5a54d970 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.519148 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-65ae5afb-e81a-4ac8-8356-d87c5a54d970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:13.524213 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 806/1610] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:13 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:13.526186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef63f79-abc2-4b1c-b9fa-f9c4815f9410 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.569504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:13.614754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef63f79-abc2-4b1c-b9fa-f9c4815f9410 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.615750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ae5afb-e81a-4ac8-8356-d87c5a54d970 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.619950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.620696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.620963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.621072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.622677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.622802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.622893 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.622974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.622974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] PATH_INFO: `/v3/users/9fa5d80ead684ce1a03bd5bdaf30920c` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.624124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9fa5d80ead684ce1a03bd5bdaf30920c)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.624225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.626098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.662619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 807/1611] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:13 2022] POST /identity/v3/projects => generated 375 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:13.662619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.664550 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:13.695510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9fa5d80ead684ce1a03bd5bdaf30920c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:13.705474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9fa5d80ead684ce1a03bd5bdaf30920c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:13.706319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9fa5d80ead684ce1a03bd5bdaf30920c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:13.706984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9fa5d80ead684ce1a03bd5bdaf30920c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:13.717365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9fa5d80ead684ce1a03bd5bdaf30920c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:13.731772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439caa16-3b4f-4bb7-b11b-6ba8d3786e36 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.738013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3e00ac-4f93-47c5-b9cc-091428e38b27 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.738685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3e00ac-4f93-47c5-b9cc-091428e38b27 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.738970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3e00ac-4f93-47c5-b9cc-091428e38b27 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.739246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3e00ac-4f93-47c5-b9cc-091428e38b27 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.741119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-fb3e00ac-4f93-47c5-b9cc-091428e38b27 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:13.742097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 805/1612] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:13 2022] DELETE /identity/v3/users/9fa5d80ead684ce1a03bd5bdaf30920c => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:13.743463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.746001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:13.816796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3e00ac-4f93-47c5-b9cc-091428e38b27 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.818607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3e00ac-4f93-47c5-b9cc-091428e38b27 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.835948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 808/1613] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:13 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:13.841409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c99b4-b17d-4e22-8fab-e85d956ea555 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.841724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c99b4-b17d-4e22-8fab-e85d956ea555 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.843753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c99b4-b17d-4e22-8fab-e85d956ea555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.860504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:13.881296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed19e1ef-cf9e-4c46-8300-ab5a802470a8 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.889807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ba987c-ebf4-45a7-8e3b-91f8b5ce506b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.890939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba987c-ebf4-45a7-8e3b-91f8b5ce506b admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.891434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba987c-ebf4-45a7-8e3b-91f8b5ce506b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.891737 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba987c-ebf4-45a7-8e3b-91f8b5ce506b admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.892321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2e6c99b4-b17d-4e22-8fab-e85d956ea555 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:13.893061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ba987c-ebf4-45a7-8e3b-91f8b5ce506b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.895414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ba987c-ebf4-45a7-8e3b-91f8b5ce506b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.896853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2e6c99b4-b17d-4e22-8fab-e85d956ea555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:13.916394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 809/1614] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:13.927428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6c99b4-b17d-4e22-8fab-e85d956ea555 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.930534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 806/1615] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:13 2022] POST /identity/v3/users => generated 288 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:13.931088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2e6c99b4-b17d-4e22-8fab-e85d956ea555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:13.935207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ba987c-ebf4-45a7-8e3b-91f8b5ce506b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.947784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ba987c-ebf4-45a7-8e3b-91f8b5ce506b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:13.953231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db95152-c15c-423c-9b51-b5490602e9b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:13.953846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db95152-c15c-423c-9b51-b5490602e9b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:13.954132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db95152-c15c-423c-9b51-b5490602e9b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:13.955013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db95152-c15c-423c-9b51-b5490602e9b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:13.955804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db95152-c15c-423c-9b51-b5490602e9b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:13.957485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db95152-c15c-423c-9b51-b5490602e9b7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:13.983249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 807/1616] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:13.984337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db95152-c15c-423c-9b51-b5490602e9b7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:13.999769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db95152-c15c-423c-9b51-b5490602e9b7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:14.005426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cbbf9b-8d1d-4657-b3d1-5d2e1bd637f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:14.006122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cbbf9b-8d1d-4657-b3d1-5d2e1bd637f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.006709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cbbf9b-8d1d-4657-b3d1-5d2e1bd637f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.007152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cbbf9b-8d1d-4657-b3d1-5d2e1bd637f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.007944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cbbf9b-8d1d-4657-b3d1-5d2e1bd637f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:14.009716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cbbf9b-8d1d-4657-b3d1-5d2e1bd637f2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:14.024440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 808/1617] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:14.028649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cbbf9b-8d1d-4657-b3d1-5d2e1bd637f2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:14.038597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6c99b4-b17d-4e22-8fab-e85d956ea555 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:14.043355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e3e7c8-2d42-478d-9801-81e85ca7280d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:14.044165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e3e7c8-2d42-478d-9801-81e85ca7280d service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.044765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e3e7c8-2d42-478d-9801-81e85ca7280d service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.045178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e3e7c8-2d42-478d-9801-81e85ca7280d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.058836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cbbf9b-8d1d-4657-b3d1-5d2e1bd637f2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:14.070064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e3e7c8-2d42-478d-9801-81e85ca7280d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:14.075789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e3e7c8-2d42-478d-9801-81e85ca7280d service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:14.078690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8220b2b-6e70-4e2b-acec-08a738cd14d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:14.081466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8220b2b-6e70-4e2b-acec-08a738cd14d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.081748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8220b2b-6e70-4e2b-acec-08a738cd14d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.082089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8220b2b-6e70-4e2b-acec-08a738cd14d7 admin admin] PATH_INFO: `/v3/projects/e9dbe6031a6747de839c3cf0b1b9bb29/users/22ac72dd70f44db0b85e8c1a6f4b068c/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.104733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 810/1618] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:14.130395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8220b2b-6e70-4e2b-acec-08a738cd14d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9dbe6031a6747de839c3cf0b1b9bb29, user_id=22ac72dd70f44db0b85e8c1a6f4b068c, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:14.136350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8220b2b-6e70-4e2b-acec-08a738cd14d7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:14.173419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 809/1619] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:14 2022] PUT /identity/v3/projects/e9dbe6031a6747de839c3cf0b1b9bb29/users/22ac72dd70f44db0b85e8c1a6f4b068c/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:14.184638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18b6479-1bae-4b40-8c44-c1e2a88c6d0e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.184927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18b6479-1bae-4b40-8c44-c1e2a88c6d0e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.185201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18b6479-1bae-4b40-8c44-c1e2a88c6d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.199701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:14.272643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a18b6479-1bae-4b40-8c44-c1e2a88c6d0e None None] MFA Rules not processed for user `22ac72dd70f44db0b85e8c1a6f4b068c`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:14.275180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a18b6479-1bae-4b40-8c44-c1e2a88c6d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:14.418812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8220b2b-6e70-4e2b-acec-08a738cd14d7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:14.426180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 811/1620] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:14.551012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8220b2b-6e70-4e2b-acec-08a738cd14d7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:14.556064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e8ac5-3785-40c0-8f6a-894f35eb41c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:14.556782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e8ac5-3785-40c0-8f6a-894f35eb41c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.557606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e8ac5-3785-40c0-8f6a-894f35eb41c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.557606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e8ac5-3785-40c0-8f6a-894f35eb41c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.558153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e8ac5-3785-40c0-8f6a-894f35eb41c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:14.560585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e8ac5-3785-40c0-8f6a-894f35eb41c0 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:14.601589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18b6479-1bae-4b40-8c44-c1e2a88c6d0e None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:14.608350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 810/1621] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:14 2022] POST /identity/v3/projects => generated 379 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:14.617243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18b6479-1bae-4b40-8c44-c1e2a88c6d0e None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:14.715116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:14.715741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.716007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.716290 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.716903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:14.719065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:14.750994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 812/1622] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:14 2022] POST /identity/v3/users => generated 296 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:14.753314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e8ac5-3785-40c0-8f6a-894f35eb41c0 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:14.783299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e8ac5-3785-40c0-8f6a-894f35eb41c0 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:14.790090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3b3566-59e0-4e60-a09f-316669147cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:14.790958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b3566-59e0-4e60-a09f-316669147cc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.794436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b3566-59e0-4e60-a09f-316669147cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.794659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b3566-59e0-4e60-a09f-316669147cc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.795378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:14.795718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3b3566-59e0-4e60-a09f-316669147cc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:14.797357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:14.798021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3b3566-59e0-4e60-a09f-316669147cc6 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:14.818206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254c229c-c2d8-42c7-9549-e5df40bb41de admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:14.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 811/1623] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:14 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:14.826125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3b3566-59e0-4e60-a09f-316669147cc6 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:14.828119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:14.828948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.830183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.830622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] PATH_INFO: `/v3/projects/9519005aee164bb0a3d0ddda970e4e81` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.833579 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] RBAC: Authorizing `identity:delete_project(project_id=9519005aee164bb0a3d0ddda970e4e81)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:14.837242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:14.838301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3b3566-59e0-4e60-a09f-316669147cc6 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:14.849363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c3dae-9fb8-4bd3-8939-b8e39bf3bd8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:14.850175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c3dae-9fb8-4bd3-8939-b8e39bf3bd8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:14.851241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c3dae-9fb8-4bd3-8939-b8e39bf3bd8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:14.851241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c3dae-9fb8-4bd3-8939-b8e39bf3bd8a admin admin] PATH_INFO: `/v3/projects/49b1a526b0e44b00986783b9434207b5/users/0a9ee94a81504c2c8bd79e8d709fcd3d/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:14.868640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673c3dae-9fb8-4bd3-8939-b8e39bf3bd8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=49b1a526b0e44b00986783b9434207b5, user_id=0a9ee94a81504c2c8bd79e8d709fcd3d, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:14.871357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673c3dae-9fb8-4bd3-8939-b8e39bf3bd8a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:14.885381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 812/1624] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:14 2022] PUT /identity/v3/projects/49b1a526b0e44b00986783b9434207b5/users/0a9ee94a81504c2c8bd79e8d709fcd3d/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:14.889583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c3dae-9fb8-4bd3-8939-b8e39bf3bd8a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:14.900351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9519005aee164bb0a3d0ddda970e4e81', 'request_id': 'req-27a3140b-2215-4cac-866f-e27ca75dc2de'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:14.907208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9519005aee164bb0a3d0ddda970e4e81', 'request_id': 'req-27a3140b-2215-4cac-866f-e27ca75dc2de'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:14.914042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9519005aee164bb0a3d0ddda970e4e81', 'request_id': 'req-27a3140b-2215-4cac-866f-e27ca75dc2de'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:14.930012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] The token cache is being invalidate because project 9519005aee164bb0a3d0ddda970e4e81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:14.930208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:14.930917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:14.933531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 813/1625] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:14 2022] DELETE /identity/v3/projects/9519005aee164bb0a3d0ddda970e4e81 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:14.936277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:14.937667 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:14.945166 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:14.954102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:14.973969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c3dae-9fb8-4bd3-8939-b8e39bf3bd8a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:15.019985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3140b-2215-4cac-866f-e27ca75dc2de admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:15.027127 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:15.028013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:15.028444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:15.028716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] PATH_INFO: `/v3/users/f73c8b191582444d8c7ecf98163faa34` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:15.029945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f73c8b191582444d8c7ecf98163faa34)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:15.031903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:15.044092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:15.044987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:15.045367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:15.045746 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:15.046516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:15.048671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:15.062168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 813/1626] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:14 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:15.066811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:15.068504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:15.081697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f5b6d2-5395-4c08-a6f8-ad90a55fb485 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:15.090516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb06cde-0af9-4455-8d95-9218166fea17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:15.090865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb06cde-0af9-4455-8d95-9218166fea17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:15.091214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb06cde-0af9-4455-8d95-9218166fea17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:15.091419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb06cde-0af9-4455-8d95-9218166fea17 admin admin] PATH_INFO: `/v3/projects/49b1a526b0e44b00986783b9434207b5/users/0a9ee94a81504c2c8bd79e8d709fcd3d/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:15.094130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb06cde-0af9-4455-8d95-9218166fea17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49b1a526b0e44b00986783b9434207b5, user_id=0a9ee94a81504c2c8bd79e8d709fcd3d, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:15.094346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f73c8b191582444d8c7ecf98163faa34'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:15.098248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb06cde-0af9-4455-8d95-9218166fea17 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:15.108892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f73c8b191582444d8c7ecf98163faa34'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:15.109668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f73c8b191582444d8c7ecf98163faa34'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:15.110317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f73c8b191582444d8c7ecf98163faa34'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:15.111666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 814/1627] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:15 2022] PUT /identity/v3/projects/49b1a526b0e44b00986783b9434207b5/users/0a9ee94a81504c2c8bd79e8d709fcd3d/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:15.119189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f73c8b191582444d8c7ecf98163faa34'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:15.122356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9127e8bc-2d96-466d-9042-d64ec6b21f47 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:15.122686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9127e8bc-2d96-466d-9042-d64ec6b21f47 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:15.122970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9127e8bc-2d96-466d-9042-d64ec6b21f47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:15.144772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 814/1628] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:14 2022] DELETE /identity/v3/users/f73c8b191582444d8c7ecf98163faa34 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:15.156983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9127e8bc-2d96-466d-9042-d64ec6b21f47 None None] MFA Rules not processed for user `0a9ee94a81504c2c8bd79e8d709fcd3d`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:15.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9127e8bc-2d96-466d-9042-d64ec6b21f47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:15.252279 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 815/1629] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:15 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:15.263799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:15.265436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:15.345764 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bc2e06-ee61-4599-9934-697f1667866d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:15.355181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:15.355822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:15.356624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:15.357033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:15.379450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:15.381144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:15.406208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 815/1630] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:15 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:15.521074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9127e8bc-2d96-466d-9042-d64ec6b21f47 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:15.535979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9127e8bc-2d96-466d-9042-d64ec6b21f47 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:15.566701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9330bd9-ac18-44e8-ac82-2c8e2d5acc4e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:15.567729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9330bd9-ac18-44e8-ac82-2c8e2d5acc4e service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:15.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9330bd9-ac18-44e8-ac82-2c8e2d5acc4e service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:15.568254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9330bd9-ac18-44e8-ac82-2c8e2d5acc4e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:15.572015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f9330bd9-ac18-44e8-ac82-2c8e2d5acc4e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:15.620472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:15.622456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:15.687613 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9330bd9-ac18-44e8-ac82-2c8e2d5acc4e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:15.691423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:15.692694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9330bd9-ac18-44e8-ac82-2c8e2d5acc4e service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:15.708396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:15.715449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 816/1631] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:15.817618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecc7c94-f228-4570-a164-f82eed30bed4 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:15.830958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:15.831431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:15.831737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:15.832006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] PATH_INFO: `/v3/projects/391b03ad8a1a4ebdb2362b59ac01cf78` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:15.835436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] RBAC: Authorizing `identity:delete_project(project_id=391b03ad8a1a4ebdb2362b59ac01cf78)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:15.840012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:15.959251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '391b03ad8a1a4ebdb2362b59ac01cf78', 'request_id': 'req-750db85a-f3fe-423a-a8f2-a6f25081e402'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:15.975321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '391b03ad8a1a4ebdb2362b59ac01cf78', 'request_id': 'req-750db85a-f3fe-423a-a8f2-a6f25081e402'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:15.985249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '391b03ad8a1a4ebdb2362b59ac01cf78', 'request_id': 'req-750db85a-f3fe-423a-a8f2-a6f25081e402'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:16.004848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] The token cache is being invalidate because project 391b03ad8a1a4ebdb2362b59ac01cf78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:16.005287 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:16.006140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-750db85a-f3fe-423a-a8f2-a6f25081e402 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:16.014318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722711f-ed35-41fc-a816-5e8b9b1a90a4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.014899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722711f-ed35-41fc-a816-5e8b9b1a90a4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.015131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722711f-ed35-41fc-a816-5e8b9b1a90a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.019342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 816/1632] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:15 2022] DELETE /identity/v3/projects/391b03ad8a1a4ebdb2362b59ac01cf78 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:16.052894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f722711f-ed35-41fc-a816-5e8b9b1a90a4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:16.054368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f722711f-ed35-41fc-a816-5e8b9b1a90a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:16.180573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 817/1633] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:16.182910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a852a123-4d7d-4231-87bc-9a323eb0ed72 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.183131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a852a123-4d7d-4231-87bc-9a323eb0ed72 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.183336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a852a123-4d7d-4231-87bc-9a323eb0ed72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.213886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a852a123-4d7d-4231-87bc-9a323eb0ed72 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:16.215306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a852a123-4d7d-4231-87bc-9a323eb0ed72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:16.228836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 817/1634] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:16.238037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f722711f-ed35-41fc-a816-5e8b9b1a90a4 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.249406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f722711f-ed35-41fc-a816-5e8b9b1a90a4 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.255289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.255880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.256162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.257123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:16.257428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.259079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.272060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 818/1635] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:16.276288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a852a123-4d7d-4231-87bc-9a323eb0ed72 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.287398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a852a123-4d7d-4231-87bc-9a323eb0ed72 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.292252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80916b6-0116-46c2-8656-7b6b5f54a94e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.292857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80916b6-0116-46c2-8656-7b6b5f54a94e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.293164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80916b6-0116-46c2-8656-7b6b5f54a94e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.293405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80916b6-0116-46c2-8656-7b6b5f54a94e admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.294049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80916b6-0116-46c2-8656-7b6b5f54a94e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.296050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80916b6-0116-46c2-8656-7b6b5f54a94e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.332586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 818/1636] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:16 2022] POST /identity/v3/projects => generated 369 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:16.336632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.348343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afff1067-6707-4649-928c-b352dcc99632 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.419147 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96db5548-9f4c-4bc8-b3d4-937f9d17d4b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.419687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96db5548-9f4c-4bc8-b3d4-937f9d17d4b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.419966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96db5548-9f4c-4bc8-b3d4-937f9d17d4b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.420239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96db5548-9f4c-4bc8-b3d4-937f9d17d4b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.420895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96db5548-9f4c-4bc8-b3d4-937f9d17d4b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.422856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96db5548-9f4c-4bc8-b3d4-937f9d17d4b4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.446006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 819/1637] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:16 2022] POST /identity/v3/users => generated 285 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:16.450275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80916b6-0116-46c2-8656-7b6b5f54a94e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.461193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80916b6-0116-46c2-8656-7b6b5f54a94e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.465679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a3a37-4656-4844-8dce-020e05b7938c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.466265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a3a37-4656-4844-8dce-020e05b7938c admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.466650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a3a37-4656-4844-8dce-020e05b7938c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.466923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a3a37-4656-4844-8dce-020e05b7938c admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.467553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83a3a37-4656-4844-8dce-020e05b7938c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.469131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83a3a37-4656-4844-8dce-020e05b7938c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.479688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 819/1638] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:16.483556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96db5548-9f4c-4bc8-b3d4-937f9d17d4b4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.493710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96db5548-9f4c-4bc8-b3d4-937f9d17d4b4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.498163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e0e3c0-9173-402e-8a45-7ef6323dc17e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.498856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e0e3c0-9173-402e-8a45-7ef6323dc17e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.499133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e0e3c0-9173-402e-8a45-7ef6323dc17e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.499403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e0e3c0-9173-402e-8a45-7ef6323dc17e admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.500041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e0e3c0-9173-402e-8a45-7ef6323dc17e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.502523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e0e3c0-9173-402e-8a45-7ef6323dc17e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.512945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 820/1639] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:16.517886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a3a37-4656-4844-8dce-020e05b7938c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.528195 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a3a37-4656-4844-8dce-020e05b7938c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.533202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385ef98c-ea8a-4246-a5ff-1e3a7497c07f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.533851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385ef98c-ea8a-4246-a5ff-1e3a7497c07f admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.534136 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385ef98c-ea8a-4246-a5ff-1e3a7497c07f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.534402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385ef98c-ea8a-4246-a5ff-1e3a7497c07f admin admin] PATH_INFO: `/v3/projects/1055ac27f9884b4fb4c615e45326e0b3/users/4613deca62d34090a469fd1c36fd8d50/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.549250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385ef98c-ea8a-4246-a5ff-1e3a7497c07f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1055ac27f9884b4fb4c615e45326e0b3, user_id=4613deca62d34090a469fd1c36fd8d50, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.551929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385ef98c-ea8a-4246-a5ff-1e3a7497c07f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.566637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 820/1640] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:16 2022] PUT /identity/v3/projects/1055ac27f9884b4fb4c615e45326e0b3/users/4613deca62d34090a469fd1c36fd8d50/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:16.577856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0597b-7640-4c80-af29-383c9deb0608 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.578207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0597b-7640-4c80-af29-383c9deb0608 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.578571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0597b-7640-4c80-af29-383c9deb0608 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.611713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-afd0597b-7640-4c80-af29-383c9deb0608 None None] MFA Rules not processed for user `4613deca62d34090a469fd1c36fd8d50`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:16.613171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-afd0597b-7640-4c80-af29-383c9deb0608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:16.674735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:16.684965 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 821/1641] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:16 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:16.690036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385ef98c-ea8a-4246-a5ff-1e3a7497c07f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.742250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:16.754822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385ef98c-ea8a-4246-a5ff-1e3a7497c07f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.759620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6923043-04fb-4525-a6da-526b396f4478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.760253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6923043-04fb-4525-a6da-526b396f4478 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.760531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6923043-04fb-4525-a6da-526b396f4478 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.760800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6923043-04fb-4525-a6da-526b396f4478 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.761421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6923043-04fb-4525-a6da-526b396f4478 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.763363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6923043-04fb-4525-a6da-526b396f4478 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.791520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 821/1642] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:16 2022] POST /identity/v3/projects => generated 369 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:16.796247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd0597b-7640-4c80-af29-383c9deb0608 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.808188 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd0597b-7640-4c80-af29-383c9deb0608 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.864474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79519415-59cb-4f99-855c-6608bae156e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.865082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79519415-59cb-4f99-855c-6608bae156e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.865361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79519415-59cb-4f99-855c-6608bae156e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.865632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79519415-59cb-4f99-855c-6608bae156e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.866257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79519415-59cb-4f99-855c-6608bae156e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.868191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79519415-59cb-4f99-855c-6608bae156e4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.891165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 822/1643] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:16 2022] POST /identity/v3/users => generated 291 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:16.895112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6923043-04fb-4525-a6da-526b396f4478 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.906606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6923043-04fb-4525-a6da-526b396f4478 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.918308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c5aec0-dc69-41a5-8255-28c704ce8399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.919001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c5aec0-dc69-41a5-8255-28c704ce8399 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.919277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c5aec0-dc69-41a5-8255-28c704ce8399 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.919595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c5aec0-dc69-41a5-8255-28c704ce8399 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.920354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c5aec0-dc69-41a5-8255-28c704ce8399 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.922049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c5aec0-dc69-41a5-8255-28c704ce8399 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:16.936016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 822/1644] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:16 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:16.940950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79519415-59cb-4f99-855c-6608bae156e4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:16.957064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79519415-59cb-4f99-855c-6608bae156e4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:16.962248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9fdf90-f509-4cc5-9996-921b4dc512c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:16.963039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9fdf90-f509-4cc5-9996-921b4dc512c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:16.963431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9fdf90-f509-4cc5-9996-921b4dc512c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:16.964020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9fdf90-f509-4cc5-9996-921b4dc512c6 admin admin] PATH_INFO: `/v3/projects/77a79fbfa1e14ca2afb123df353e9d6e/users/3093f5c953ae43899a083cf58b32c8c2/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:16.984106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9fdf90-f509-4cc5-9996-921b4dc512c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77a79fbfa1e14ca2afb123df353e9d6e, user_id=3093f5c953ae43899a083cf58b32c8c2, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:16.987630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9fdf90-f509-4cc5-9996-921b4dc512c6 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:17.003246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 823/1645] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:16 2022] PUT /identity/v3/projects/77a79fbfa1e14ca2afb123df353e9d6e/users/3093f5c953ae43899a083cf58b32c8c2/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:17.007742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c5aec0-dc69-41a5-8255-28c704ce8399 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:17.019213 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c5aec0-dc69-41a5-8255-28c704ce8399 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:17.088957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf180ab0-d536-41f3-93de-fa6bdb1cf850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:17.091211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf180ab0-d536-41f3-93de-fa6bdb1cf850 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:17.091704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf180ab0-d536-41f3-93de-fa6bdb1cf850 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:17.092091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf180ab0-d536-41f3-93de-fa6bdb1cf850 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:17.092760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf180ab0-d536-41f3-93de-fa6bdb1cf850 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:17.095980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf180ab0-d536-41f3-93de-fa6bdb1cf850 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:17.111994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 823/1646] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:17 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:17.113740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9fdf90-f509-4cc5-9996-921b4dc512c6 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:17.124694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9fdf90-f509-4cc5-9996-921b4dc512c6 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:17.130980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:17.131976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:17.132439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:17.132746 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] PATH_INFO: `/v3/projects/77a79fbfa1e14ca2afb123df353e9d6e/users/3093f5c953ae43899a083cf58b32c8c2/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:17.149206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] RBAC: Authorizing `identity:create_grant(project_id=77a79fbfa1e14ca2afb123df353e9d6e, user_id=3093f5c953ae43899a083cf58b32c8c2, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:17.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:17.165142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 824/1647] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:17 2022] PUT /identity/v3/projects/77a79fbfa1e14ca2afb123df353e9d6e/users/3093f5c953ae43899a083cf58b32c8c2/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:17.181991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d1d46-e4c2-411b-abc1-0bad09894f9d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:17.182544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d1d46-e4c2-411b-abc1-0bad09894f9d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:17.182858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d1d46-e4c2-411b-abc1-0bad09894f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:17.213903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-bf3d1d46-e4c2-411b-abc1-0bad09894f9d None None] MFA Rules not processed for user `3093f5c953ae43899a083cf58b32c8c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:17.215294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bf3d1d46-e4c2-411b-abc1-0bad09894f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:17.291026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 824/1648] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:17 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:17.305892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:17.308446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:17.374431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba3f385-81ca-4b9a-9d45-2d802d70317c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:17.380727 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cc2ba9-aa24-4e68-bb54-19d5560f9ea4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:17.381509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc2ba9-aa24-4e68-bb54-19d5560f9ea4 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:17.381909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc2ba9-aa24-4e68-bb54-19d5560f9ea4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:17.382283 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc2ba9-aa24-4e68-bb54-19d5560f9ea4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:17.393566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cc2ba9-aa24-4e68-bb54-19d5560f9ea4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:17.395344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cc2ba9-aa24-4e68-bb54-19d5560f9ea4 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:17.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 825/1649] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:17 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:18.851407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3d1d46-e4c2-411b-abc1-0bad09894f9d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:18.900674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:18.916939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3d1d46-e4c2-411b-abc1-0bad09894f9d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:18.921831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:18.922508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:18.922792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:18.923887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] PATH_INFO: `/v3/users/4613deca62d34090a469fd1c36fd8d50` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:18.925024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4613deca62d34090a469fd1c36fd8d50)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:18.926982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:18.988946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4613deca62d34090a469fd1c36fd8d50'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:19.008761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4613deca62d34090a469fd1c36fd8d50'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:19.008761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4613deca62d34090a469fd1c36fd8d50'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:19.011567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4613deca62d34090a469fd1c36fd8d50'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:19.028413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4613deca62d34090a469fd1c36fd8d50'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:19.067264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 825/1650] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:18 2022] DELETE /identity/v3/users/4613deca62d34090a469fd1c36fd8d50 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:19.071972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc3ff7-d5a7-489d-b28a-b710afa9c040 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:19.072462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc3ff7-d5a7-489d-b28a-b710afa9c040 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:19.073088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc3ff7-d5a7-489d-b28a-b710afa9c040 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:19.094631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ecfc3ff7-d5a7-489d-b28a-b710afa9c040 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:19.096788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ecfc3ff7-d5a7-489d-b28a-b710afa9c040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:19.151915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:19.166922 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 826/1651] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:19.177504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:19.178980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:19.252387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da24b829-07ac-46a2-a1cd-82d001100005 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:19.256603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d0e1a6-bfa1-4981-8209-a19ddaeacdc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:19.257328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0e1a6-bfa1-4981-8209-a19ddaeacdc0 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:19.257707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0e1a6-bfa1-4981-8209-a19ddaeacdc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:19.258083 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0e1a6-bfa1-4981-8209-a19ddaeacdc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:19.269369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d0e1a6-bfa1-4981-8209-a19ddaeacdc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:19.271108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d0e1a6-bfa1-4981-8209-a19ddaeacdc0 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:19.281479 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 826/1652] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:20.000020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfc3ff7-d5a7-489d-b28a-b710afa9c040 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:20.001602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ecfc3ff7-d5a7-489d-b28a-b710afa9c040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:20.022381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfc3ff7-d5a7-489d-b28a-b710afa9c040 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:20.029168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:20.032659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:20.033114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:20.033424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] PATH_INFO: `/v3/projects/1055ac27f9884b4fb4c615e45326e0b3` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:20.035171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=1055ac27f9884b4fb4c615e45326e0b3)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:20.037561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:20.121134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1055ac27f9884b4fb4c615e45326e0b3', 'request_id': 'req-9417037f-f56f-459e-bfbf-39d35c3550aa'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.139176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1055ac27f9884b4fb4c615e45326e0b3', 'request_id': 'req-9417037f-f56f-459e-bfbf-39d35c3550aa'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.153720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1055ac27f9884b4fb4c615e45326e0b3', 'request_id': 'req-9417037f-f56f-459e-bfbf-39d35c3550aa'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.169754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] The token cache is being invalidate because project 1055ac27f9884b4fb4c615e45326e0b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:20.170314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.171367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.173456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 827/1653] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:19 2022] DELETE /identity/v3/projects/1055ac27f9884b4fb4c615e45326e0b3 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:20.178325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d0e1a6-bfa1-4981-8209-a19ddaeacdc0 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:20.199559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d0e1a6-bfa1-4981-8209-a19ddaeacdc0 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:20.310588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:20.311894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:20.313645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:20.314277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] PATH_INFO: `/v3/users/3093f5c953ae43899a083cf58b32c8c2` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:20.317380 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3093f5c953ae43899a083cf58b32c8c2)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:20.320721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:20.347994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:20.402651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3093f5c953ae43899a083cf58b32c8c2'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.416323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3093f5c953ae43899a083cf58b32c8c2'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.417173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3093f5c953ae43899a083cf58b32c8c2'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.417764 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3093f5c953ae43899a083cf58b32c8c2'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.427308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1ffb7558-7fa8-4265-9b89-25952952e120 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3093f5c953ae43899a083cf58b32c8c2'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.462852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 827/1654] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:20 2022] DELETE /identity/v3/users/3093f5c953ae43899a083cf58b32c8c2 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:20.800792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:20.802288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:20.819718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9417037f-f56f-459e-bfbf-39d35c3550aa admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:20.901630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:20.902285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:20.902595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:20.902898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] PATH_INFO: `/v3/projects/77a79fbfa1e14ca2afb123df353e9d6e` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:20.904214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] RBAC: Authorizing `identity:delete_project(project_id=77a79fbfa1e14ca2afb123df353e9d6e)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:20.906331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:20.971041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77a79fbfa1e14ca2afb123df353e9d6e', 'request_id': 'req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.979193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77a79fbfa1e14ca2afb123df353e9d6e', 'request_id': 'req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:20.988772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77a79fbfa1e14ca2afb123df353e9d6e', 'request_id': 'req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:21.003939 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] The token cache is being invalidate because project 77a79fbfa1e14ca2afb123df353e9d6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:21.004361 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:21.005069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dc03ca25-ae71-4635-bab1-7456b8ba8ccf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:21.006756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 828/1655] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:20 2022] DELETE /identity/v3/projects/77a79fbfa1e14ca2afb123df353e9d6e => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:21.011777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe868d5-f6e8-4615-8964-5803d7919d20 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.012284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe868d5-f6e8-4615-8964-5803d7919d20 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.012596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe868d5-f6e8-4615-8964-5803d7919d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.043768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-cfe868d5-f6e8-4615-8964-5803d7919d20 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:21.045038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-cfe868d5-f6e8-4615-8964-5803d7919d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:21.117776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 828/1656] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:21.132438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186eae1-aac1-47f6-87c2-04d45b73757f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.133015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186eae1-aac1-47f6-87c2-04d45b73757f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.133148 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186eae1-aac1-47f6-87c2-04d45b73757f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.147207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:21.163574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-4186eae1-aac1-47f6-87c2-04d45b73757f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:21.165504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4186eae1-aac1-47f6-87c2-04d45b73757f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:21.176053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 829/1657] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:21.181363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe868d5-f6e8-4615-8964-5803d7919d20 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.196618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe868d5-f6e8-4615-8964-5803d7919d20 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.203905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186eae1-aac1-47f6-87c2-04d45b73757f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.204647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.205340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.205613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.205875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.206374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4186eae1-aac1-47f6-87c2-04d45b73757f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:21.206566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:21.206867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.208465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.219218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186eae1-aac1-47f6-87c2-04d45b73757f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.222384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 829/1658] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:21.225528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.226167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.226586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.226861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] PATH_INFO: `/v3/users/d85356c8535d4065b846272a1b559e43` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.226957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.227887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d85356c8535d4065b846272a1b559e43)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.229857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.237276 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82552c2e-7c6d-486f-b791-14932875925f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.242146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979d962a-6515-4000-ac2c-6493880701b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.242767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d962a-6515-4000-ac2c-6493880701b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.243046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d962a-6515-4000-ac2c-6493880701b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.243312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d962a-6515-4000-ac2c-6493880701b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.243937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979d962a-6515-4000-ac2c-6493880701b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.245933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979d962a-6515-4000-ac2c-6493880701b3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.285539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 830/1659] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:21 2022] POST /identity/v3/projects => generated 385 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:21.285689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd85356c8535d4065b846272a1b559e43'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:21.290976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979d962a-6515-4000-ac2c-6493880701b3 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.293696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd85356c8535d4065b846272a1b559e43'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:21.294678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd85356c8535d4065b846272a1b559e43'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:21.295354 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd85356c8535d4065b846272a1b559e43'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:21.306219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-107b6aac-56f0-4921-937e-6f34da24abf2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd85356c8535d4065b846272a1b559e43'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:21.331937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 830/1660] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:21 2022] DELETE /identity/v3/users/d85356c8535d4065b846272a1b559e43 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:21.336541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77ffa2-1d42-4d7c-8066-870f9f703661 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.336830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77ffa2-1d42-4d7c-8066-870f9f703661 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.337350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77ffa2-1d42-4d7c-8066-870f9f703661 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.360200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ec77ffa2-1d42-4d7c-8066-870f9f703661 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:21.361483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ec77ffa2-1d42-4d7c-8066-870f9f703661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:21.370693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:21.377911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979d962a-6515-4000-ac2c-6493880701b3 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.446657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 831/1661] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:21.454847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec77ffa2-1d42-4d7c-8066-870f9f703661 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.456662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ec77ffa2-1d42-4d7c-8066-870f9f703661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:21.466167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.467005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.467385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.467749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.469048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.471376 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.499375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 831/1662] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:21 2022] POST /identity/v3/users => generated 293 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:21.505212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.508430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:21.526415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c4ce71-4ac5-4952-aa28-2cee3c723416 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.533422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fc2d9b-0c41-44a0-a945-b54ce36febc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.534041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc2d9b-0c41-44a0-a945-b54ce36febc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.534322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc2d9b-0c41-44a0-a945-b54ce36febc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.534626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc2d9b-0c41-44a0-a945-b54ce36febc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.535272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fc2d9b-0c41-44a0-a945-b54ce36febc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.536875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fc2d9b-0c41-44a0-a945-b54ce36febc4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.547265 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec77ffa2-1d42-4d7c-8066-870f9f703661 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.550102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 832/1663] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:21 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:21.553668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fc2d9b-0c41-44a0-a945-b54ce36febc4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.558002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf7b9eb-3ae3-48e0-a420-17b5817f43f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.558518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7b9eb-3ae3-48e0-a420-17b5817f43f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.558950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7b9eb-3ae3-48e0-a420-17b5817f43f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.559290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7b9eb-3ae3-48e0-a420-17b5817f43f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.565278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fc2d9b-0c41-44a0-a945-b54ce36febc4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.570157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf7b9eb-3ae3-48e0-a420-17b5817f43f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.571736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf7b9eb-3ae3-48e0-a420-17b5817f43f4 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.576410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c1122a-28a6-4165-a086-e1549427a7f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.577693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c1122a-28a6-4165-a086-e1549427a7f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.578032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c1122a-28a6-4165-a086-e1549427a7f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.578373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c1122a-28a6-4165-a086-e1549427a7f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.579184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c1122a-28a6-4165-a086-e1549427a7f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.581432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c1122a-28a6-4165-a086-e1549427a7f3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.582909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 832/1664] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:21.597195 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 833/1665] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:21 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:21.602898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf7b9eb-3ae3-48e0-a420-17b5817f43f4 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.613274 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf7b9eb-3ae3-48e0-a420-17b5817f43f4 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.618108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba260aa4-ce6f-447d-9ff8-84182c6abcc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.618805 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba260aa4-ce6f-447d-9ff8-84182c6abcc4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.619084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba260aa4-ce6f-447d-9ff8-84182c6abcc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.619520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba260aa4-ce6f-447d-9ff8-84182c6abcc4 admin admin] PATH_INFO: `/v3/projects/9c333c7fd6534efd8cb510608e5deb09/users/9155821aa20c4500975d70e63d02c668/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.632724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:21.637639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba260aa4-ce6f-447d-9ff8-84182c6abcc4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c333c7fd6534efd8cb510608e5deb09, user_id=9155821aa20c4500975d70e63d02c668, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.640979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba260aa4-ce6f-447d-9ff8-84182c6abcc4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.655933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 833/1666] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:21 2022] PUT /identity/v3/projects/9c333c7fd6534efd8cb510608e5deb09/users/9155821aa20c4500975d70e63d02c668/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:21.668119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c97670-7fec-477a-a681-a82c048f0e7b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.668519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c97670-7fec-477a-a681-a82c048f0e7b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.668870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c97670-7fec-477a-a681-a82c048f0e7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.698614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-85c97670-7fec-477a-a681-a82c048f0e7b None None] MFA Rules not processed for user `9155821aa20c4500975d70e63d02c668`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:21.700383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-85c97670-7fec-477a-a681-a82c048f0e7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:21.781978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 834/1667] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:21.787331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba260aa4-ce6f-447d-9ff8-84182c6abcc4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.863233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba260aa4-ce6f-447d-9ff8-84182c6abcc4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:21.869519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f9b9f4-a475-4edd-8eed-d3655b31efb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:21.870204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9b9f4-a475-4edd-8eed-d3655b31efb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:21.870662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9b9f4-a475-4edd-8eed-d3655b31efb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:21.871271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9b9f4-a475-4edd-8eed-d3655b31efb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:21.872034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f9b9f4-a475-4edd-8eed-d3655b31efb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:21.874293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f9b9f4-a475-4edd-8eed-d3655b31efb2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:21.905870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c97670-7fec-477a-a681-a82c048f0e7b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.908079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-85c97670-7fec-477a-a681-a82c048f0e7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:21.910343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 834/1668] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:21 2022] POST /identity/v3/projects => generated 385 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:21.914716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f9b9f4-a475-4edd-8eed-d3655b31efb2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:21.945539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c97670-7fec-477a-a681-a82c048f0e7b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:22.063608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:22.064586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:22.065202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:22.065619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] PATH_INFO: `/v3/projects/8b6e57e5385c44959ce32df7da918bf8` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:22.068427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b6e57e5385c44959ce32df7da918bf8)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:22.070600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f9b9f4-a475-4edd-8eed-d3655b31efb2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:22.071960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:22.076555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:22.086910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fca76c1-99d5-48dc-81f4-84583061e141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:22.088945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca76c1-99d5-48dc-81f4-84583061e141 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:22.091014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca76c1-99d5-48dc-81f4-84583061e141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:22.091313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fca76c1-99d5-48dc-81f4-84583061e141 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:22.093332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fca76c1-99d5-48dc-81f4-84583061e141 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:22.113307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fca76c1-99d5-48dc-81f4-84583061e141 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:22.156228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b6e57e5385c44959ce32df7da918bf8', 'request_id': 'req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.169184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 835/1669] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:21 2022] POST /identity/v3/users => generated 299 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:22.175356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fca76c1-99d5-48dc-81f4-84583061e141 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:22.177305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b6e57e5385c44959ce32df7da918bf8', 'request_id': 'req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.191504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b6e57e5385c44959ce32df7da918bf8', 'request_id': 'req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.197839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fca76c1-99d5-48dc-81f4-84583061e141 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:22.219071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] The token cache is being invalidate because project 8b6e57e5385c44959ce32df7da918bf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:22.219071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.219071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.219071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 835/1670] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:21 2022] DELETE /identity/v3/projects/8b6e57e5385c44959ce32df7da918bf8 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:22.228397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:22.236606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:22.333301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:22.342916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:22.343224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:22.343506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:22.344265 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:22.348348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:22.364446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:22.365878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5afb8c-0e56-4a1e-b870-c67fbb884a56 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:22.372721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:22.373823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:22.374112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:22.374388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] PATH_INFO: `/v3/users/22ac72dd70f44db0b85e8c1a6f4b068c` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:22.375688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=22ac72dd70f44db0b85e8c1a6f4b068c)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:22.381862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:22.382450 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 836/1671] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:22.383566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:22.385650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:22.408049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e869807-00cf-446d-a804-f74d667b5bb8 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:22.414352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23acb80f-0ae9-41f9-a167-9c138377cad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:22.415178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23acb80f-0ae9-41f9-a167-9c138377cad8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:22.415522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23acb80f-0ae9-41f9-a167-9c138377cad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:22.416193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23acb80f-0ae9-41f9-a167-9c138377cad8 admin admin] PATH_INFO: `/v3/projects/4ab8558ef2ce470c85995159106c38ab/users/d6fecbccafc7416998df2313feec968c/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:22.436716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23acb80f-0ae9-41f9-a167-9c138377cad8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ab8558ef2ce470c85995159106c38ab, user_id=d6fecbccafc7416998df2313feec968c, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:22.439925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23acb80f-0ae9-41f9-a167-9c138377cad8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:22.457872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 837/1672] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:22 2022] PUT /identity/v3/projects/4ab8558ef2ce470c85995159106c38ab/users/d6fecbccafc7416998df2313feec968c/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:22.462511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23acb80f-0ae9-41f9-a167-9c138377cad8 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:22.465999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '22ac72dd70f44db0b85e8c1a6f4b068c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.481162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '22ac72dd70f44db0b85e8c1a6f4b068c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.482431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '22ac72dd70f44db0b85e8c1a6f4b068c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.483106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '22ac72dd70f44db0b85e8c1a6f4b068c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.496204 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '22ac72dd70f44db0b85e8c1a6f4b068c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:22.535568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 836/1673] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:22 2022] DELETE /identity/v3/users/22ac72dd70f44db0b85e8c1a6f4b068c => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:22.589828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23acb80f-0ae9-41f9-a167-9c138377cad8 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:22.687185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d401b5-a2d0-480e-b617-7690219d3d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:22.688181 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d401b5-a2d0-480e-b617-7690219d3d00 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:22.689394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d401b5-a2d0-480e-b617-7690219d3d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:22.689918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d401b5-a2d0-480e-b617-7690219d3d00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:22.690804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d401b5-a2d0-480e-b617-7690219d3d00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:22.693172 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d401b5-a2d0-480e-b617-7690219d3d00 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:22.708569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 838/1674] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:22.712974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:22.715238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:22.752676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49bc6e-ef46-44f7-863f-f5de4f94dc6b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:22.763084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:22.764205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:22.764685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:22.765125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] PATH_INFO: `/v3/projects/4ab8558ef2ce470c85995159106c38ab/users/d6fecbccafc7416998df2313feec968c/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:22.768271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ab8558ef2ce470c85995159106c38ab, user_id=d6fecbccafc7416998df2313feec968c, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:22.772013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:22.783872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:22.804081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 837/1675] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:22 2022] PUT /identity/v3/projects/4ab8558ef2ce470c85995159106c38ab/users/d6fecbccafc7416998df2313feec968c/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:22.818089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd22df5-84a1-43fb-9a3a-51b5b0859395 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:22.818821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd22df5-84a1-43fb-9a3a-51b5b0859395 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:22.819466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd22df5-84a1-43fb-9a3a-51b5b0859395 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:22.868017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-dcd22df5-84a1-43fb-9a3a-51b5b0859395 None None] MFA Rules not processed for user `d6fecbccafc7416998df2313feec968c`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:22.869902 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-dcd22df5-84a1-43fb-9a3a-51b5b0859395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:22.980983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 839/1676] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:22 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:22.995470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:22.997435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:23.070858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd705ad1-4a31-4733-95a1-1dbba730ffea admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:23.074853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:23.075493 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:23.075863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:23.076180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:23.088818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:23.090460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:23.104541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 838/1677] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:22 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:23.107484 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd22df5-84a1-43fb-9a3a-51b5b0859395 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:23.109488 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-dcd22df5-84a1-43fb-9a3a-51b5b0859395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:23.131185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:23.174343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:23.175834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:23.266052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd22df5-84a1-43fb-9a3a-51b5b0859395 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:23.271882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:23.273585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:23.273585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:23.273585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] PATH_INFO: `/v3/projects/e9dbe6031a6747de839c3cf0b1b9bb29` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:23.274196 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9dbe6031a6747de839c3cf0b1b9bb29)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:23.276146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:23.301435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc7ba87-731e-4609-a5df-49f0231ce01c service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:23.310624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:23.311225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:23.311509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:23.312102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] PATH_INFO: `/v3/users/9480d0455a214c55adeeaee9a2eace8f` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:23.313310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9480d0455a214c55adeeaee9a2eace8f)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:23.316644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:23.383905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9dbe6031a6747de839c3cf0b1b9bb29', 'request_id': 'req-418fab89-ac0e-45f4-98b1-7ded74267554'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.399162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9480d0455a214c55adeeaee9a2eace8f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.400088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9dbe6031a6747de839c3cf0b1b9bb29', 'request_id': 'req-418fab89-ac0e-45f4-98b1-7ded74267554'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.412926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9480d0455a214c55adeeaee9a2eace8f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.413724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9dbe6031a6747de839c3cf0b1b9bb29', 'request_id': 'req-418fab89-ac0e-45f4-98b1-7ded74267554'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.414847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9480d0455a214c55adeeaee9a2eace8f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.416082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9480d0455a214c55adeeaee9a2eace8f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.432654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] The token cache is being invalidate because project e9dbe6031a6747de839c3cf0b1b9bb29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:23.433170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.433487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9480d0455a214c55adeeaee9a2eace8f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.434892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.436921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 840/1678] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:23 2022] DELETE /identity/v3/projects/e9dbe6031a6747de839c3cf0b1b9bb29 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:23.437878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:23.439657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:23.460626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 839/1679] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:23 2022] DELETE /identity/v3/users/9480d0455a214c55adeeaee9a2eace8f => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:23.461482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:23.465884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:23.501588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:23.577907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da968f24-a3a6-4bc2-8205-60e4fbb9a8d7 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:23.581778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418fab89-ac0e-45f4-98b1-7ded74267554 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:23.585669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:23.586915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:23.587517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:23.587861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] PATH_INFO: `/v3/users/0a9ee94a81504c2c8bd79e8d709fcd3d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:23.589229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a9ee94a81504c2c8bd79e8d709fcd3d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:23.591241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:23.591396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:23.591895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:23.592183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:23.592464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] PATH_INFO: `/v3/users/9155821aa20c4500975d70e63d02c668` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:23.593781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9155821aa20c4500975d70e63d02c668)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:23.596082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:23.656525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:23.669491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9155821aa20c4500975d70e63d02c668'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.671327 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a9ee94a81504c2c8bd79e8d709fcd3d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.680029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9155821aa20c4500975d70e63d02c668'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.680295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a9ee94a81504c2c8bd79e8d709fcd3d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.681087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a9ee94a81504c2c8bd79e8d709fcd3d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.681286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9155821aa20c4500975d70e63d02c668'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.681716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a9ee94a81504c2c8bd79e8d709fcd3d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.682210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9155821aa20c4500975d70e63d02c668'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.690979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-342db6aa-2547-46b3-baf3-d0233a3c8802 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a9ee94a81504c2c8bd79e8d709fcd3d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.694408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-58ebbedd-4409-49ad-bbff-17dd011973f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9155821aa20c4500975d70e63d02c668'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:23.713354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 840/1680] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:23 2022] DELETE /identity/v3/users/0a9ee94a81504c2c8bd79e8d709fcd3d => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:23.715098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af19af39-1727-4bab-a72b-59967ae8188c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:23.715643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af19af39-1727-4bab-a72b-59967ae8188c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:23.715953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af19af39-1727-4bab-a72b-59967ae8188c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:23.728076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 841/1681] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:23 2022] DELETE /identity/v3/users/9155821aa20c4500975d70e63d02c668 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:23.732727 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8377d5-b47a-4ce3-9fb8-44c9bae36313 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:23.733015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8377d5-b47a-4ce3-9fb8-44c9bae36313 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:23.733295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8377d5-b47a-4ce3-9fb8-44c9bae36313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:23.743419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-af19af39-1727-4bab-a72b-59967ae8188c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:23.744724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-af19af39-1727-4bab-a72b-59967ae8188c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:23.770649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ee8377d5-b47a-4ce3-9fb8-44c9bae36313 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:23.772013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ee8377d5-b47a-4ce3-9fb8-44c9bae36313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:23.909339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 841/1682] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:23.921550 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 842/1683] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:23.944719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af19af39-1727-4bab-a72b-59967ae8188c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:23.946667 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-af19af39-1727-4bab-a72b-59967ae8188c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:23.967218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8377d5-b47a-4ce3-9fb8-44c9bae36313 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:23.968843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ee8377d5-b47a-4ce3-9fb8-44c9bae36313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:24.039805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:24.084015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af19af39-1727-4bab-a72b-59967ae8188c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:24.095959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:24.096933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:24.098466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:24.098916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:24.106756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8377d5-b47a-4ce3-9fb8-44c9bae36313 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:24.120582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b313804-1812-434c-9519-614d22ea6886 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:24.121452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b313804-1812-434c-9519-614d22ea6886 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:24.121918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b313804-1812-434c-9519-614d22ea6886 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:24.122371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b313804-1812-434c-9519-614d22ea6886 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:24.123693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:24.129122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:24.146560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b313804-1812-434c-9519-614d22ea6886 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:24.148726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 842/1684] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:24.153146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b313804-1812-434c-9519-614d22ea6886 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:24.169167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 843/1685] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:24.543392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:24.546182 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:24.564222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4ab5bc-0eef-4fca-987f-c7a08b710cc9 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:24.569303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:24.569960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:24.570516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:24.570903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] PATH_INFO: `/v3/projects/49b1a526b0e44b00986783b9434207b5` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:24.572036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] RBAC: Authorizing `identity:delete_project(project_id=49b1a526b0e44b00986783b9434207b5)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:24.574074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:24.635065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49b1a526b0e44b00986783b9434207b5', 'request_id': 'req-3a1bf104-08e3-4893-b289-d43e31f9f227'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:24.643896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49b1a526b0e44b00986783b9434207b5', 'request_id': 'req-3a1bf104-08e3-4893-b289-d43e31f9f227'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:24.651700 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49b1a526b0e44b00986783b9434207b5', 'request_id': 'req-3a1bf104-08e3-4893-b289-d43e31f9f227'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:24.674507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] The token cache is being invalidate because project 49b1a526b0e44b00986783b9434207b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:24.675623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:24.676613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:24.679007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 843/1686] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:24 2022] DELETE /identity/v3/projects/49b1a526b0e44b00986783b9434207b5 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:24.685213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b9c62c-3d22-4271-82ac-7425807085c4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:24.685682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b9c62c-3d22-4271-82ac-7425807085c4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:24.685769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b9c62c-3d22-4271-82ac-7425807085c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:24.730758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f4b9c62c-3d22-4271-82ac-7425807085c4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:24.730758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f4b9c62c-3d22-4271-82ac-7425807085c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:24.788344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:24.789954 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:24.931225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:24.952196 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 844/1687] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:24.962034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c28879-ec4b-44d6-80af-6c7aa7544c20 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:24.962531 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c28879-ec4b-44d6-80af-6c7aa7544c20 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:24.962999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c28879-ec4b-44d6-80af-6c7aa7544c20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:24.965688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1bf104-08e3-4893-b289-d43e31f9f227 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:24.982835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:24.983529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:24.983808 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:24.984081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] PATH_INFO: `/v3/projects/f7ceb8b35c74456f92354d47e1802e54` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:24.985305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7ceb8b35c74456f92354d47e1802e54)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:24.987618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:24.990639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-65c28879-ec4b-44d6-80af-6c7aa7544c20 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:24.992770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-65c28879-ec4b-44d6-80af-6c7aa7544c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:25.008800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 845/1688] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:25.010213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c28879-ec4b-44d6-80af-6c7aa7544c20 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.031160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c28879-ec4b-44d6-80af-6c7aa7544c20 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.042132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.042853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.043122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.043402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.044066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:25.044377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.053471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.072003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 846/1689] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:25.080243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.095739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7ceb8b35c74456f92354d47e1802e54', 'request_id': 'req-8af5bcb5-3f34-4a48-8f87-70da828be191'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.108573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8a6903-ab15-41e7-bba0-d8811b06e84f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.114724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7ceb8b35c74456f92354d47e1802e54', 'request_id': 'req-8af5bcb5-3f34-4a48-8f87-70da828be191'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.145341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7ceb8b35c74456f92354d47e1802e54', 'request_id': 'req-8af5bcb5-3f34-4a48-8f87-70da828be191'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.153081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:25.196081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] The token cache is being invalidate because project f7ceb8b35c74456f92354d47e1802e54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:25.196248 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.202087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.203801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 844/1690] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:24 2022] DELETE /identity/v3/projects/f7ceb8b35c74456f92354d47e1802e54 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:25.208960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.213823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:25.312702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2681af93-86dc-413f-afea-638f6a0d2fb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.315264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681af93-86dc-413f-afea-638f6a0d2fb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.315765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681af93-86dc-413f-afea-638f6a0d2fb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.316118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681af93-86dc-413f-afea-638f6a0d2fb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.316848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2681af93-86dc-413f-afea-638f6a0d2fb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.326975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2681af93-86dc-413f-afea-638f6a0d2fb4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.332025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af5bcb5-3f34-4a48-8f87-70da828be191 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.338282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.339708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.340245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.340702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] PATH_INFO: `/v3/projects/9c333c7fd6534efd8cb510608e5deb09` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.342856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c333c7fd6534efd8cb510608e5deb09)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.345227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.372075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 847/1691] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:25 2022] POST /identity/v3/projects => generated 377 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:25.377299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5e1e48-ef0c-4c68-b1f5-0399bea38fd2 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.377541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5e1e48-ef0c-4c68-b1f5-0399bea38fd2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.377752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5e1e48-ef0c-4c68-b1f5-0399bea38fd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.406973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c333c7fd6534efd8cb510608e5deb09', 'request_id': 'req-8c35f536-6805-4c8e-a5d0-42ce2e553246'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.415952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c333c7fd6534efd8cb510608e5deb09', 'request_id': 'req-8c35f536-6805-4c8e-a5d0-42ce2e553246'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.425005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-7f5e1e48-ef0c-4c68-b1f5-0399bea38fd2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:25.426905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7f5e1e48-ef0c-4c68-b1f5-0399bea38fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:25.431358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c333c7fd6534efd8cb510608e5deb09', 'request_id': 'req-8c35f536-6805-4c8e-a5d0-42ce2e553246'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.458116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] The token cache is being invalidate because project 9c333c7fd6534efd8cb510608e5deb09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:25.459846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.461366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.464231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 845/1692] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:25 2022] DELETE /identity/v3/projects/9c333c7fd6534efd8cb510608e5deb09 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:25.465753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.469177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:25.510269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 848/1693] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:25.512023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5e1e48-ef0c-4c68-b1f5-0399bea38fd2 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.514381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7f5e1e48-ef0c-4c68-b1f5-0399bea38fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:25.528321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5e1e48-ef0c-4c68-b1f5-0399bea38fd2 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.535349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.536004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.536381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.536750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] PATH_INFO: `/v3/users/d6fecbccafc7416998df2313feec968c` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.538744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6fecbccafc7416998df2313feec968c)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.541124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.617298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6fecbccafc7416998df2313feec968c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.622078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c35f536-6805-4c8e-a5d0-42ce2e553246 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.628947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b792f4-32c3-4068-ae8a-40e449c0ebdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.629798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b792f4-32c3-4068-ae8a-40e449c0ebdb admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.630255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b792f4-32c3-4068-ae8a-40e449c0ebdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.630714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b792f4-32c3-4068-ae8a-40e449c0ebdb admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.631430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6fecbccafc7416998df2313feec968c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.631833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b792f4-32c3-4068-ae8a-40e449c0ebdb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.634263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b792f4-32c3-4068-ae8a-40e449c0ebdb admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.636717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6fecbccafc7416998df2313feec968c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.638979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6fecbccafc7416998df2313feec968c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.643595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:25.659440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6fecbccafc7416998df2313feec968c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:25.669945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 846/1694] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:25 2022] POST /identity/v3/users => generated 289 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:25.671578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823aa2b0-b809-4dbf-ab3b-85442ee20dfa None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.672145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823aa2b0-b809-4dbf-ab3b-85442ee20dfa None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.672678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823aa2b0-b809-4dbf-ab3b-85442ee20dfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.695807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 849/1695] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:25 2022] DELETE /identity/v3/users/d6fecbccafc7416998df2313feec968c => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:25.695807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.695807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:25.707614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-823aa2b0-b809-4dbf-ab3b-85442ee20dfa None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:25.709284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-823aa2b0-b809-4dbf-ab3b-85442ee20dfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:25.721136 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:25.818724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 847/1696] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:25.823558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823aa2b0-b809-4dbf-ab3b-85442ee20dfa None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.842255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823aa2b0-b809-4dbf-ab3b-85442ee20dfa None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.842804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31d0e8d-f77c-41fb-9805-26c0ab459521 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.852015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.853651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.854207 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.854791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.855370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd01482d-99bb-4ee5-b3ae-86af23e4526e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.856249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:25.856910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.857006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd01482d-99bb-4ee5-b3ae-86af23e4526e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.857298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd01482d-99bb-4ee5-b3ae-86af23e4526e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.857695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd01482d-99bb-4ee5-b3ae-86af23e4526e admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.858358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd01482d-99bb-4ee5-b3ae-86af23e4526e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.860241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd01482d-99bb-4ee5-b3ae-86af23e4526e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.861236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.877617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 850/1697] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:25.880165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd01482d-99bb-4ee5-b3ae-86af23e4526e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.884395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 848/1698] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:25.890562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.893963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd01482d-99bb-4ee5-b3ae-86af23e4526e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.905250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136bbe4b-e9bd-49ae-8807-0efdf8f2fdc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.906146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136bbe4b-e9bd-49ae-8807-0efdf8f2fdc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.906460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136bbe4b-e9bd-49ae-8807-0efdf8f2fdc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.906760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136bbe4b-e9bd-49ae-8807-0efdf8f2fdc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.907722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136bbe4b-e9bd-49ae-8807-0efdf8f2fdc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.908503 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3402306-fa5b-4393-bb4e-ab91ee7939a9 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.912884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136bbe4b-e9bd-49ae-8807-0efdf8f2fdc7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.914566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9256b7ee-6349-402f-89a6-44450ba020a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.915516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256b7ee-6349-402f-89a6-44450ba020a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.916439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256b7ee-6349-402f-89a6-44450ba020a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.916810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256b7ee-6349-402f-89a6-44450ba020a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.918077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9256b7ee-6349-402f-89a6-44450ba020a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:25.920237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9256b7ee-6349-402f-89a6-44450ba020a6 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:25.932734 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 851/1699] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:25.937009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136bbe4b-e9bd-49ae-8807-0efdf8f2fdc7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:25.952278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136bbe4b-e9bd-49ae-8807-0efdf8f2fdc7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:25.979174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd753f5-ce07-4c5f-bb1b-31cbbea8e662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:25.980083 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd753f5-ce07-4c5f-bb1b-31cbbea8e662 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:25.980531 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd753f5-ce07-4c5f-bb1b-31cbbea8e662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:25.980822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd753f5-ce07-4c5f-bb1b-31cbbea8e662 admin admin] PATH_INFO: `/v3/projects/e8ddf006ee284515ba78fa0280e2d498/users/e0336be78e4a45209b090cd2fe056915/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:25.984871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 849/1700] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:25 2022] POST /identity/v3/projects => generated 375 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:25.988940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9256b7ee-6349-402f-89a6-44450ba020a6 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:26.001691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd753f5-ce07-4c5f-bb1b-31cbbea8e662 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8ddf006ee284515ba78fa0280e2d498, user_id=e0336be78e4a45209b090cd2fe056915, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:26.005287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd753f5-ce07-4c5f-bb1b-31cbbea8e662 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:26.027183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 852/1701] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:25 2022] PUT /identity/v3/projects/e8ddf006ee284515ba78fa0280e2d498/users/e0336be78e4a45209b090cd2fe056915/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:26.040788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5663420d-cd90-486a-9db6-10609765394f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5663420d-cd90-486a-9db6-10609765394f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.041348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5663420d-cd90-486a-9db6-10609765394f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:26.079639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5663420d-cd90-486a-9db6-10609765394f None None] MFA Rules not processed for user `e0336be78e4a45209b090cd2fe056915`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:26.081112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5663420d-cd90-486a-9db6-10609765394f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:26.130699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9256b7ee-6349-402f-89a6-44450ba020a6 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:26.226542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 853/1702] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:26.240376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5663420d-cd90-486a-9db6-10609765394f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:26.268013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc88331-742a-49f4-b0f3-19cefc1db1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:26.268580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc88331-742a-49f4-b0f3-19cefc1db1fb admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.268864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc88331-742a-49f4-b0f3-19cefc1db1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.269137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc88331-742a-49f4-b0f3-19cefc1db1fb admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:26.269783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc88331-742a-49f4-b0f3-19cefc1db1fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:26.272919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc88331-742a-49f4-b0f3-19cefc1db1fb admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:26.324638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:26.361943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 850/1703] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:25 2022] POST /identity/v3/users => generated 288 bytes in 373 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:26.365963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc88331-742a-49f4-b0f3-19cefc1db1fb admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:26.392495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc88331-742a-49f4-b0f3-19cefc1db1fb admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:26.421651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b253a0-a3a2-4639-a01c-c5ff7cecd04d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:26.422287 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b253a0-a3a2-4639-a01c-c5ff7cecd04d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.422598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b253a0-a3a2-4639-a01c-c5ff7cecd04d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.423504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b253a0-a3a2-4639-a01c-c5ff7cecd04d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:26.424234 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b253a0-a3a2-4639-a01c-c5ff7cecd04d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:26.426092 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b253a0-a3a2-4639-a01c-c5ff7cecd04d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:26.449456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 851/1704] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:26.453356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b253a0-a3a2-4639-a01c-c5ff7cecd04d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:26.460686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5663420d-cd90-486a-9db6-10609765394f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:26.473224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77635590-379c-47be-99d5-05a74192c68d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:26.473224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77635590-379c-47be-99d5-05a74192c68d admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.473224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77635590-379c-47be-99d5-05a74192c68d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.473224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77635590-379c-47be-99d5-05a74192c68d admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:26.473912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77635590-379c-47be-99d5-05a74192c68d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:26.475880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77635590-379c-47be-99d5-05a74192c68d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:26.480033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:26.485543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b253a0-a3a2-4639-a01c-c5ff7cecd04d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:26.504115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeaaba4-89c7-4f98-90eb-bf9355a553e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:26.505336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeaaba4-89c7-4f98-90eb-bf9355a553e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.507229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeaaba4-89c7-4f98-90eb-bf9355a553e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.508435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeaaba4-89c7-4f98-90eb-bf9355a553e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:26.509846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfeaaba4-89c7-4f98-90eb-bf9355a553e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:26.514666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfeaaba4-89c7-4f98-90eb-bf9355a553e1 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:26.536516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 852/1705] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:26.540577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeaaba4-89c7-4f98-90eb-bf9355a553e1 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:26.551031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 854/1706] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:26 2022] POST /identity/v3/projects => generated 375 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:26.552963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77635590-379c-47be-99d5-05a74192c68d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:26.562520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeaaba4-89c7-4f98-90eb-bf9355a553e1 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:26.697114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77635590-379c-47be-99d5-05a74192c68d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:26.710071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:26.712237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.712672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.713135 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:26.713855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:26.716315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:26.734704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bd5029-cc68-4bf9-8288-af2bb5637486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:26.736105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bd5029-cc68-4bf9-8288-af2bb5637486 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.736555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bd5029-cc68-4bf9-8288-af2bb5637486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.737094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bd5029-cc68-4bf9-8288-af2bb5637486 admin admin] PATH_INFO: `/v3/projects/7786db617b14422da7b9e7117eeffdca/users/f6123a9aff334f6d95c8b3e49a63f441/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:26.748530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 855/1707] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:26 2022] POST /identity/v3/users => generated 288 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:26.750319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:26.753238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:26.759179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bd5029-cc68-4bf9-8288-af2bb5637486 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7786db617b14422da7b9e7117eeffdca, user_id=f6123a9aff334f6d95c8b3e49a63f441, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:26.761874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bd5029-cc68-4bf9-8288-af2bb5637486 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:26.769800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a208fb35-2cc3-444b-9b46-e3b56ef3c77c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:26.776326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:26.777258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.777764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.778277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] PATH_INFO: `/v3/projects/4ab8558ef2ce470c85995159106c38ab` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:26.779680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ab8558ef2ce470c85995159106c38ab)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:26.781941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:26.797444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 853/1708] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:26 2022] PUT /identity/v3/projects/7786db617b14422da7b9e7117eeffdca/users/f6123a9aff334f6d95c8b3e49a63f441/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:26.797444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bd5029-cc68-4bf9-8288-af2bb5637486 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:26.903463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ab8558ef2ce470c85995159106c38ab', 'request_id': 'req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:26.913628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bd5029-cc68-4bf9-8288-af2bb5637486 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:26.921116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ab8558ef2ce470c85995159106c38ab', 'request_id': 'req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:26.933908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ab8558ef2ce470c85995159106c38ab', 'request_id': 'req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:26.965657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] The token cache is being invalidate because project 4ab8558ef2ce470c85995159106c38ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:26.965840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:26.966564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-87a44f38-3cc5-46e5-9ffa-8b0fd7c0e90a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:26.968504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 856/1709] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:26 2022] DELETE /identity/v3/projects/4ab8558ef2ce470c85995159106c38ab => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:26.969760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e597c9-d019-467d-bcbc-a734f90d8f82 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:26.970214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e597c9-d019-467d-bcbc-a734f90d8f82 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:26.971068 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e597c9-d019-467d-bcbc-a734f90d8f82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.018619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6726aa07-9766-4975-b9fe-a3f58fff0d5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.019971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6726aa07-9766-4975-b9fe-a3f58fff0d5a admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.020408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6726aa07-9766-4975-b9fe-a3f58fff0d5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.021006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6726aa07-9766-4975-b9fe-a3f58fff0d5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.021999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6726aa07-9766-4975-b9fe-a3f58fff0d5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.024006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6726aa07-9766-4975-b9fe-a3f58fff0d5a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.038180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 854/1710] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:27.039717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73888b8b-5f21-4218-89a8-17e4e790c46e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.040006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73888b8b-5f21-4218-89a8-17e4e790c46e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.040279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73888b8b-5f21-4218-89a8-17e4e790c46e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.045071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-09e597c9-d019-467d-bcbc-a734f90d8f82 None None] MFA Rules not processed for user `f6123a9aff334f6d95c8b3e49a63f441`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:27.046308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-09e597c9-d019-467d-bcbc-a734f90d8f82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:27.061514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:27.067465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-73888b8b-5f21-4218-89a8-17e4e790c46e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:27.068704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-73888b8b-5f21-4218-89a8-17e4e790c46e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:27.086814 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 855/1711] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:27.087788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73888b8b-5f21-4218-89a8-17e4e790c46e None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.089737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-73888b8b-5f21-4218-89a8-17e4e790c46e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:27.104232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73888b8b-5f21-4218-89a8-17e4e790c46e None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.110081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7d807f-067e-4982-9d3c-a36cd30026f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.112174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7d807f-067e-4982-9d3c-a36cd30026f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.112636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7d807f-067e-4982-9d3c-a36cd30026f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.113167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7d807f-067e-4982-9d3c-a36cd30026f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.114147 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7d807f-067e-4982-9d3c-a36cd30026f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.115847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7d807f-067e-4982-9d3c-a36cd30026f1 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.127193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 856/1712] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:27.128588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50653593-e4f5-4d48-98df-939ac9158315 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.128876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50653593-e4f5-4d48-98df-939ac9158315 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.129149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50653593-e4f5-4d48-98df-939ac9158315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.140475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:27.167098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-50653593-e4f5-4d48-98df-939ac9158315 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:27.168338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-50653593-e4f5-4d48-98df-939ac9158315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:27.175725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 857/1713] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:26 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:27.180371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e597c9-d019-467d-bcbc-a734f90d8f82 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.190413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 857/1714] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:27.191339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50653593-e4f5-4d48-98df-939ac9158315 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.193349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-50653593-e4f5-4d48-98df-939ac9158315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:27.221144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e597c9-d019-467d-bcbc-a734f90d8f82 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.226158 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e495d9-573e-429b-866b-086a647968a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.226951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e495d9-573e-429b-866b-086a647968a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.227233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e495d9-573e-429b-866b-086a647968a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.227510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e495d9-573e-429b-866b-086a647968a1 admin admin] PATH_INFO: `/v3/projects/c738162d881d4baa934897639f60a2d4/users/d46c96dc74ff4956bf4a1f83261bcaaa/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.256205 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e495d9-573e-429b-866b-086a647968a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c738162d881d4baa934897639f60a2d4, user_id=d46c96dc74ff4956bf4a1f83261bcaaa, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.259064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e495d9-573e-429b-866b-086a647968a1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.279508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 858/1715] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:27 2022] PUT /identity/v3/projects/c738162d881d4baa934897639f60a2d4/users/d46c96dc74ff4956bf4a1f83261bcaaa/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:27.280713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e495d9-573e-429b-866b-086a647968a1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.288112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50653593-e4f5-4d48-98df-939ac9158315 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.357441 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd794cd9-39db-416f-b99c-281554293984 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.358134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd794cd9-39db-416f-b99c-281554293984 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.358422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd794cd9-39db-416f-b99c-281554293984 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.358728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd794cd9-39db-416f-b99c-281554293984 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.379373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e495d9-573e-429b-866b-086a647968a1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.386709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21250706-768a-4329-8660-d53f37d74b6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.387695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21250706-768a-4329-8660-d53f37d74b6e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.388179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21250706-768a-4329-8660-d53f37d74b6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.388552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21250706-768a-4329-8660-d53f37d74b6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.389328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21250706-768a-4329-8660-d53f37d74b6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:27.390995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21250706-768a-4329-8660-d53f37d74b6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.392260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21250706-768a-4329-8660-d53f37d74b6e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.413256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 859/1716] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:27.414797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6bccc-79b2-4819-a316-6270d2c8bdd0 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.416723 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6bccc-79b2-4819-a316-6270d2c8bdd0 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.417241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6bccc-79b2-4819-a316-6270d2c8bdd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.435742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd794cd9-39db-416f-b99c-281554293984 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.437523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd794cd9-39db-416f-b99c-281554293984 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.448409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 858/1717] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:27 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:27.449218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd794cd9-39db-416f-b99c-281554293984 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.462261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd794cd9-39db-416f-b99c-281554293984 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.467389 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d39b1b-f0d5-4c49-a8ad-3bc7ae388505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.467924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-42b6bccc-79b2-4819-a316-6270d2c8bdd0 None None] MFA Rules not processed for user `d46c96dc74ff4956bf4a1f83261bcaaa`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:27.468052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d39b1b-f0d5-4c49-a8ad-3bc7ae388505 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.468360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d39b1b-f0d5-4c49-a8ad-3bc7ae388505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.468685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d39b1b-f0d5-4c49-a8ad-3bc7ae388505 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.469225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-42b6bccc-79b2-4819-a316-6270d2c8bdd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:27.469382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d39b1b-f0d5-4c49-a8ad-3bc7ae388505 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.472063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d39b1b-f0d5-4c49-a8ad-3bc7ae388505 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.519151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 859/1718] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:27 2022] POST /identity/v3/projects => generated 391 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:27.522925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d39b1b-f0d5-4c49-a8ad-3bc7ae388505 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.583426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 860/1719] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:27 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:27.588084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b6bccc-79b2-4819-a316-6270d2c8bdd0 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.600137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b6bccc-79b2-4819-a316-6270d2c8bdd0 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.610549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:27.631343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:27.659922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d39b1b-f0d5-4c49-a8ad-3bc7ae388505 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.669336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9074138d-cad0-4cc5-bc0b-2509a39c8b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.669969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9074138d-cad0-4cc5-bc0b-2509a39c8b2e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.670982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9074138d-cad0-4cc5-bc0b-2509a39c8b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.671611 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9074138d-cad0-4cc5-bc0b-2509a39c8b2e admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.672297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9074138d-cad0-4cc5-bc0b-2509a39c8b2e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.675348 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9074138d-cad0-4cc5-bc0b-2509a39c8b2e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.683332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418528a9-588d-4ca5-a59b-2a0c1c000a58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.684020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418528a9-588d-4ca5-a59b-2a0c1c000a58 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.684302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418528a9-588d-4ca5-a59b-2a0c1c000a58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.684577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418528a9-588d-4ca5-a59b-2a0c1c000a58 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.685787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418528a9-588d-4ca5-a59b-2a0c1c000a58 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.687308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418528a9-588d-4ca5-a59b-2a0c1c000a58 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.721410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 860/1720] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:27 2022] POST /identity/v3/users => generated 296 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:27.725227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9074138d-cad0-4cc5-bc0b-2509a39c8b2e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.742078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9074138d-cad0-4cc5-bc0b-2509a39c8b2e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.746840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 861/1721] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:27 2022] POST /identity/v3/projects => generated 375 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:27.747379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f8b76-474a-45ad-b158-38c147167cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.748080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f8b76-474a-45ad-b158-38c147167cd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.748551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f8b76-474a-45ad-b158-38c147167cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.749011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f8b76-474a-45ad-b158-38c147167cd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.749679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418528a9-588d-4ca5-a59b-2a0c1c000a58 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.749793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f8b76-474a-45ad-b158-38c147167cd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.753821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f8b76-474a-45ad-b158-38c147167cd3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.778155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 861/1722] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:27.779798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f8b76-474a-45ad-b158-38c147167cd3 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.809841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f8b76-474a-45ad-b158-38c147167cd3 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.904877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418528a9-588d-4ca5-a59b-2a0c1c000a58 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.918793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7df0d38-63b6-4cdc-9118-a060dd9d61ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.919813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7df0d38-63b6-4cdc-9118-a060dd9d61ef admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.920242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7df0d38-63b6-4cdc-9118-a060dd9d61ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.920382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e79d6f-b67a-401e-9f1e-906203b3e8f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.920810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7df0d38-63b6-4cdc-9118-a060dd9d61ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.921062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e79d6f-b67a-401e-9f1e-906203b3e8f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.921324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e79d6f-b67a-401e-9f1e-906203b3e8f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.921537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e79d6f-b67a-401e-9f1e-906203b3e8f2 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.922183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e79d6f-b67a-401e-9f1e-906203b3e8f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.923108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7df0d38-63b6-4cdc-9118-a060dd9d61ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:27.925195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e79d6f-b67a-401e-9f1e-906203b3e8f2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.925605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7df0d38-63b6-4cdc-9118-a060dd9d61ef admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:27.943679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 862/1723] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:27.945024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7df0d38-63b6-4cdc-9118-a060dd9d61ef admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:27.959820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7df0d38-63b6-4cdc-9118-a060dd9d61ef admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:27.965311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b46a261-785f-4d8a-b39b-ce5d1f6778cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:27.965997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b46a261-785f-4d8a-b39b-ce5d1f6778cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:27.966256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b46a261-785f-4d8a-b39b-ce5d1f6778cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:27.966617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b46a261-785f-4d8a-b39b-ce5d1f6778cf admin admin] PATH_INFO: `/v3/projects/88a8af88dac0499fa667717053378616/users/e0a9d27def4542ebb5b766b6ba9145bd/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:27.986722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 862/1724] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:27 2022] POST /identity/v3/users => generated 294 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:27.994899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e79d6f-b67a-401e-9f1e-906203b3e8f2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.007991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b46a261-785f-4d8a-b39b-ce5d1f6778cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=88a8af88dac0499fa667717053378616, user_id=e0a9d27def4542ebb5b766b6ba9145bd, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.010789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b46a261-785f-4d8a-b39b-ce5d1f6778cf admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.012561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e79d6f-b67a-401e-9f1e-906203b3e8f2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.018767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755702f8-313b-4aac-bd90-5ed9f2216113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.019437 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755702f8-313b-4aac-bd90-5ed9f2216113 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.019724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755702f8-313b-4aac-bd90-5ed9f2216113 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.020023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755702f8-313b-4aac-bd90-5ed9f2216113 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.020804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755702f8-313b-4aac-bd90-5ed9f2216113 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.022592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755702f8-313b-4aac-bd90-5ed9f2216113 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.028493 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 863/1725] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:27 2022] PUT /identity/v3/projects/88a8af88dac0499fa667717053378616/users/e0a9d27def4542ebb5b766b6ba9145bd/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:28.036241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 863/1726] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:28.037892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b46a261-785f-4d8a-b39b-ce5d1f6778cf admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.042959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f8e7a-a5fb-4097-9d8c-b6c6e9cc521c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.043243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f8e7a-a5fb-4097-9d8c-b6c6e9cc521c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.043515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f8e7a-a5fb-4097-9d8c-b6c6e9cc521c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.082818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-fe7f8e7a-a5fb-4097-9d8c-b6c6e9cc521c None None] MFA Rules not processed for user `e0a9d27def4542ebb5b766b6ba9145bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:28.084316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fe7f8e7a-a5fb-4097-9d8c-b6c6e9cc521c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:28.119154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b46a261-785f-4d8a-b39b-ce5d1f6778cf admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.124967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105173ac-d2d2-437a-a75d-24c10ea62980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.126372 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105173ac-d2d2-437a-a75d-24c10ea62980 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.126837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105173ac-d2d2-437a-a75d-24c10ea62980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.127222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105173ac-d2d2-437a-a75d-24c10ea62980 admin admin] PATH_INFO: `/v3/projects/d6ceda4660fc49e3b8d9010acf282cdc/users/2c8c63a0e580407f94977a3a0e6ca01f/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.146250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105173ac-d2d2-437a-a75d-24c10ea62980 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6ceda4660fc49e3b8d9010acf282cdc, user_id=2c8c63a0e580407f94977a3a0e6ca01f, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.150256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105173ac-d2d2-437a-a75d-24c10ea62980 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.174685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 864/1727] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:28 2022] PUT /identity/v3/projects/d6ceda4660fc49e3b8d9010acf282cdc/users/2c8c63a0e580407f94977a3a0e6ca01f/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:28.181372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 864/1728] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:28 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:28.181521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105173ac-d2d2-437a-a75d-24c10ea62980 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.182349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7f8e7a-a5fb-4097-9d8c-b6c6e9cc521c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.196591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7f8e7a-a5fb-4097-9d8c-b6c6e9cc521c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.201333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:28.207754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:28.274224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105173ac-d2d2-437a-a75d-24c10ea62980 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.279099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e943738b-991b-47e9-bcb1-47399f087fbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.280013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943738b-991b-47e9-bcb1-47399f087fbd admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.280365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943738b-991b-47e9-bcb1-47399f087fbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.280760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e943738b-991b-47e9-bcb1-47399f087fbd admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.281179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33ab5de-8eef-4730-a77c-ec72a6879202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.281556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e943738b-991b-47e9-bcb1-47399f087fbd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ab5de-8eef-4730-a77c-ec72a6879202 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.283258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ab5de-8eef-4730-a77c-ec72a6879202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.283706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ab5de-8eef-4730-a77c-ec72a6879202 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.284446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e943738b-991b-47e9-bcb1-47399f087fbd admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.284802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33ab5de-8eef-4730-a77c-ec72a6879202 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.286634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33ab5de-8eef-4730-a77c-ec72a6879202 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.301593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 865/1729] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:28.306017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33ab5de-8eef-4730-a77c-ec72a6879202 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.321587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33ab5de-8eef-4730-a77c-ec72a6879202 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.325778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 865/1730] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:28 2022] POST /identity/v3/projects => generated 391 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:28.329556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ddb206-bf64-4e69-901f-d93e3798b286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.330377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e943738b-991b-47e9-bcb1-47399f087fbd admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.331074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ddb206-bf64-4e69-901f-d93e3798b286 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.331594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ddb206-bf64-4e69-901f-d93e3798b286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.332073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ddb206-bf64-4e69-901f-d93e3798b286 admin admin] PATH_INFO: `/v3/projects/d6ceda4660fc49e3b8d9010acf282cdc/users/2c8c63a0e580407f94977a3a0e6ca01f/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.334552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ddb206-bf64-4e69-901f-d93e3798b286 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6ceda4660fc49e3b8d9010acf282cdc, user_id=2c8c63a0e580407f94977a3a0e6ca01f, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.338179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ddb206-bf64-4e69-901f-d93e3798b286 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.362509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 866/1731] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:28 2022] PUT /identity/v3/projects/d6ceda4660fc49e3b8d9010acf282cdc/users/2c8c63a0e580407f94977a3a0e6ca01f/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:28.364452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fe3c88-f717-42fa-91ea-360380a0f8ea None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.364968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fe3c88-f717-42fa-91ea-360380a0f8ea None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.365495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fe3c88-f717-42fa-91ea-360380a0f8ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.414056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-42fe3c88-f717-42fa-91ea-360380a0f8ea None None] MFA Rules not processed for user `2c8c63a0e580407f94977a3a0e6ca01f`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-42fe3c88-f717-42fa-91ea-360380a0f8ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:28.418521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e943738b-991b-47e9-bcb1-47399f087fbd admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.525248 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 867/1732] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:28 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:28.561045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fe3c88-f717-42fa-91ea-360380a0f8ea None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.563225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f0b708-1e53-45b5-8477-72c02158ddd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.564424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f0b708-1e53-45b5-8477-72c02158ddd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.564707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f0b708-1e53-45b5-8477-72c02158ddd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.564982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f0b708-1e53-45b5-8477-72c02158ddd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.565621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f0b708-1e53-45b5-8477-72c02158ddd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.567596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f0b708-1e53-45b5-8477-72c02158ddd7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.600625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 866/1733] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:28 2022] POST /identity/v3/users => generated 302 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:28.602940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f0b708-1e53-45b5-8477-72c02158ddd7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.636914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f0b708-1e53-45b5-8477-72c02158ddd7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.642214 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5ddf0c-e5b6-4d27-8f46-34f061df6d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.642869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5ddf0c-e5b6-4d27-8f46-34f061df6d43 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.643398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5ddf0c-e5b6-4d27-8f46-34f061df6d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.643685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5ddf0c-e5b6-4d27-8f46-34f061df6d43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.644336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5ddf0c-e5b6-4d27-8f46-34f061df6d43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.645956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5ddf0c-e5b6-4d27-8f46-34f061df6d43 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.670630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 867/1734] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:28.675246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5ddf0c-e5b6-4d27-8f46-34f061df6d43 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.686320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fe3c88-f717-42fa-91ea-360380a0f8ea None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.689549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5ddf0c-e5b6-4d27-8f46-34f061df6d43 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.692196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.693755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.694880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f3cff-b3dd-45de-ba3a-7adfc5300ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.695519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.696701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.696999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f3cff-b3dd-45de-ba3a-7adfc5300ff1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.697297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f3cff-b3dd-45de-ba3a-7adfc5300ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.697583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f3cff-b3dd-45de-ba3a-7adfc5300ff1 admin admin] PATH_INFO: `/v3/projects/937f86c5cbb0457f905b637785c31f28/users/bbb565c99521478dbdb5615b7bf9d280/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.699594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:28.716860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f3cff-b3dd-45de-ba3a-7adfc5300ff1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=937f86c5cbb0457f905b637785c31f28, user_id=bbb565c99521478dbdb5615b7bf9d280, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.720970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f3cff-b3dd-45de-ba3a-7adfc5300ff1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.724629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:28.742102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 868/1735] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:28 2022] PUT /identity/v3/projects/937f86c5cbb0457f905b637785c31f28/users/bbb565c99521478dbdb5615b7bf9d280/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:28.746621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f3cff-b3dd-45de-ba3a-7adfc5300ff1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.755724 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:28.779454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.781443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.806260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 868/1736] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:28 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:28.845811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f3cff-b3dd-45de-ba3a-7adfc5300ff1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.859448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f3cddc-301e-4198-91fc-82705dc5c14b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.861059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f3cddc-301e-4198-91fc-82705dc5c14b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.861668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f3cddc-301e-4198-91fc-82705dc5c14b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.861946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f3cddc-301e-4198-91fc-82705dc5c14b admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.862634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f3cddc-301e-4198-91fc-82705dc5c14b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.865049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f3cddc-301e-4198-91fc-82705dc5c14b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.892516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:28.893934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 869/1737] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:28.919164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a894f27-b640-49d8-ad9a-d5a0b67b1024 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:28.936256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a17bf7-383e-4d36-b9b9-f90a544e7119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:28.937279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a17bf7-383e-4d36-b9b9-f90a544e7119 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.937718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a17bf7-383e-4d36-b9b9-f90a544e7119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.938751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a17bf7-383e-4d36-b9b9-f90a544e7119 admin admin] PATH_INFO: `/v3/projects/937f86c5cbb0457f905b637785c31f28/users/bbb565c99521478dbdb5615b7bf9d280/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:28.944242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a17bf7-383e-4d36-b9b9-f90a544e7119 admin admin] RBAC: Authorizing `identity:create_grant(project_id=937f86c5cbb0457f905b637785c31f28, user_id=bbb565c99521478dbdb5615b7bf9d280, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:28.950092 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a17bf7-383e-4d36-b9b9-f90a544e7119 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:28.989755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 869/1738] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:28 2022] PUT /identity/v3/projects/937f86c5cbb0457f905b637785c31f28/users/bbb565c99521478dbdb5615b7bf9d280/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:28.992196 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88e8fb-4b5e-4bdf-98e6-8cab24c2ec82 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:28.992627 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88e8fb-4b5e-4bdf-98e6-8cab24c2ec82 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:28.992981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88e8fb-4b5e-4bdf-98e6-8cab24c2ec82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:29.033982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-fb88e8fb-4b5e-4bdf-98e6-8cab24c2ec82 None None] MFA Rules not processed for user `bbb565c99521478dbdb5615b7bf9d280`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:29.035633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fb88e8fb-4b5e-4bdf-98e6-8cab24c2ec82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:29.198842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 870/1739] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:28 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:29.233296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a17bf7-383e-4d36-b9b9-f90a544e7119 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:29.344620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a17bf7-383e-4d36-b9b9-f90a544e7119 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:29.349889 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f60e2e4-3ee2-436b-94a6-860a86572644 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:29.350618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f60e2e4-3ee2-436b-94a6-860a86572644 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:29.351050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f60e2e4-3ee2-436b-94a6-860a86572644 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:29.351331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f60e2e4-3ee2-436b-94a6-860a86572644 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:29.366093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f60e2e4-3ee2-436b-94a6-860a86572644 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:29.368574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f60e2e4-3ee2-436b-94a6-860a86572644 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:29.385632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 870/1740] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:29 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:29.455145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb88e8fb-4b5e-4bdf-98e6-8cab24c2ec82 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:29.456427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fb88e8fb-4b5e-4bdf-98e6-8cab24c2ec82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:29.475022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb88e8fb-4b5e-4bdf-98e6-8cab24c2ec82 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:29.479670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a83db62-4597-40cb-8a3b-579039a2207a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:29.480682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a83db62-4597-40cb-8a3b-579039a2207a service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:29.481784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a83db62-4597-40cb-8a3b-579039a2207a service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:29.482208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a83db62-4597-40cb-8a3b-579039a2207a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:29.593110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a83db62-4597-40cb-8a3b-579039a2207a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:29.597802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a83db62-4597-40cb-8a3b-579039a2207a service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:29.613218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 871/1741] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:29 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:30.352883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1464e81-2195-4bd0-8a5d-1b06acf59bc4 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:30.353472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1464e81-2195-4bd0-8a5d-1b06acf59bc4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:30.353921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1464e81-2195-4bd0-8a5d-1b06acf59bc4 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:30.355673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 871/1742] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:41:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:41:30.366096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1ed9f-0c4e-40cd-956b-63999be8b2d6 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:30.366730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1ed9f-0c4e-40cd-956b-63999be8b2d6 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:30.366811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1ed9f-0c4e-40cd-956b-63999be8b2d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:30.388994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-49d1ed9f-0c4e-40cd-956b-63999be8b2d6 None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:30.390779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-49d1ed9f-0c4e-40cd-956b-63999be8b2d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:30.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:30.451035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1464e81-2195-4bd0-8a5d-1b06acf59bc4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:30.453297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c1464e81-2195-4bd0-8a5d-1b06acf59bc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:30.520048 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 872/1743] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:41:30 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:30.533926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d1ed9f-0c4e-40cd-956b-63999be8b2d6 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:30.536155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-49d1ed9f-0c4e-40cd-956b-63999be8b2d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:30.560174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d1ed9f-0c4e-40cd-956b-63999be8b2d6 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:30.569748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7fa36-bded-4030-83ee-87f30df88a58 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:30.570906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7fa36-bded-4030-83ee-87f30df88a58 service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:30.571419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7fa36-bded-4030-83ee-87f30df88a58 service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:30.571937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7fa36-bded-4030-83ee-87f30df88a58 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:30.582503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee7fa36-bded-4030-83ee-87f30df88a58 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:30.583369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1464e81-2195-4bd0-8a5d-1b06acf59bc4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:30.584126 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee7fa36-bded-4030-83ee-87f30df88a58 service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:30.589038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:30.589715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:30.590000 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:30.590267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] PATH_INFO: `/v3/users/0e0407fdc07f4ad98d09fdabbc9952fa` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:30.591653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e0407fdc07f4ad98d09fdabbc9952fa)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:30.593645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:30.595624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 873/1744] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:30 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:30.665228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e0407fdc07f4ad98d09fdabbc9952fa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:30.672450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e0407fdc07f4ad98d09fdabbc9952fa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:30.673316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e0407fdc07f4ad98d09fdabbc9952fa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:30.673950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e0407fdc07f4ad98d09fdabbc9952fa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:30.681978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e0407fdc07f4ad98d09fdabbc9952fa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:30.691303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:30.698151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7fa36-bded-4030-83ee-87f30df88a58 service placement] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:30.709391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7fa36-bded-4030-83ee-87f30df88a58 service placement] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:30.712341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 872/1745] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:30 2022] DELETE /identity/v3/users/0e0407fdc07f4ad98d09fdabbc9952fa => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:30.715259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:30.716364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:30.716643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:30.716922 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:30.718817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:30.789959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:30.791544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:30.801841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 874/1746] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:30 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:30.909427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:30.911125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:30.994039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c37b85e-9212-4206-9e44-993cb1a3c32c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:30.999597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:31.000249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.000529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.000947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] PATH_INFO: `/v3/projects/0e082898df784f5c8eaa55f0cc717de5` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.002640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e082898df784f5c8eaa55f0cc717de5)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:31.005824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:31.066470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e082898df784f5c8eaa55f0cc717de5', 'request_id': 'req-4b576783-0d43-4340-b0a1-dd15e2909824'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.073651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e082898df784f5c8eaa55f0cc717de5', 'request_id': 'req-4b576783-0d43-4340-b0a1-dd15e2909824'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.080812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e082898df784f5c8eaa55f0cc717de5', 'request_id': 'req-4b576783-0d43-4340-b0a1-dd15e2909824'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.093921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] The token cache is being invalidate because project 0e082898df784f5c8eaa55f0cc717de5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:31.094210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.094996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.096903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 873/1747] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:30 2022] DELETE /identity/v3/projects/0e082898df784f5c8eaa55f0cc717de5 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:31.102701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:31.115579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae333b8-fba4-46af-b245-d5079e65113a service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:31.182555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:31.183282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.183481 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] PATH_INFO: `/v3/users/c837648810924b9ea593e43bc8cd5ba8` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.184914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c837648810924b9ea593e43bc8cd5ba8)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:31.186887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:31.238179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:31.256544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c837648810924b9ea593e43bc8cd5ba8'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.266117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c837648810924b9ea593e43bc8cd5ba8'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.267892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c837648810924b9ea593e43bc8cd5ba8'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.268474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c837648810924b9ea593e43bc8cd5ba8'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.277526 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c837648810924b9ea593e43bc8cd5ba8'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.300578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 875/1748] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:31 2022] DELETE /identity/v3/users/c837648810924b9ea593e43bc8cd5ba8 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:31.400997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:31.402529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:31.441768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b576783-0d43-4340-b0a1-dd15e2909824 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:31.565950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:31.566550 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.566834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.567202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] PATH_INFO: `/v3/users/f6123a9aff334f6d95c8b3e49a63f441` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.568428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6123a9aff334f6d95c8b3e49a63f441)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:31.570468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:31.577624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:31.579896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:31.593671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81623391-2693-4dda-9642-e62e8f943305 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:31.599563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:31.600601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.600772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.601096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] PATH_INFO: `/v3/projects/178860dc773c4926b01b6a8e639662ea` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.602315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] RBAC: Authorizing `identity:delete_project(project_id=178860dc773c4926b01b6a8e639662ea)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:31.604328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:31.611064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:31.644870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6123a9aff334f6d95c8b3e49a63f441'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.652632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6123a9aff334f6d95c8b3e49a63f441'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.653438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6123a9aff334f6d95c8b3e49a63f441'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.654045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6123a9aff334f6d95c8b3e49a63f441'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.662349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6123a9aff334f6d95c8b3e49a63f441'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.662812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '178860dc773c4926b01b6a8e639662ea', 'request_id': 'req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.668658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '178860dc773c4926b01b6a8e639662ea', 'request_id': 'req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.677000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '178860dc773c4926b01b6a8e639662ea', 'request_id': 'req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.685580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 874/1749] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:31 2022] DELETE /identity/v3/users/f6123a9aff334f6d95c8b3e49a63f441 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:31.685580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:31.685580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:31.692566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] The token cache is being invalidate because project 178860dc773c4926b01b6a8e639662ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:31.693307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.697780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b5070e91-3569-4a5f-9cbd-59cbbd4b5f60 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:31.703635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 876/1750] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:31 2022] DELETE /identity/v3/projects/178860dc773c4926b01b6a8e639662ea => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:31.703635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56eb3e5-c431-46a8-9cf4-99d90f07b7a8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.703635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56eb3e5-c431-46a8-9cf4-99d90f07b7a8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.703635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56eb3e5-c431-46a8-9cf4-99d90f07b7a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.725547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a56eb3e5-c431-46a8-9cf4-99d90f07b7a8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:31.727334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a56eb3e5-c431-46a8-9cf4-99d90f07b7a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:31.761010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43f226-0839-403f-8750-a98ed061b9da admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:31.766020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92e09bc-dee5-49e2-9d4d-ba1c54787b8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:31.766752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e09bc-dee5-49e2-9d4d-ba1c54787b8f service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.767037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e09bc-dee5-49e2-9d4d-ba1c54787b8f service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.767312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e09bc-dee5-49e2-9d4d-ba1c54787b8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.769625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f92e09bc-dee5-49e2-9d4d-ba1c54787b8f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:31.804391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 877/1751] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:31.811508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167034a-ea22-4743-b468-21157342692e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.812016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167034a-ea22-4743-b468-21157342692e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.812474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167034a-ea22-4743-b468-21157342692e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.836373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-6167034a-ea22-4743-b468-21157342692e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:31.837827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6167034a-ea22-4743-b468-21157342692e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:31.842024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92e09bc-dee5-49e2-9d4d-ba1c54787b8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:31.843493 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92e09bc-dee5-49e2-9d4d-ba1c54787b8f service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:31.850745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 878/1752] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:31.850745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6167034a-ea22-4743-b468-21157342692e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:31.850745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6167034a-ea22-4743-b468-21157342692e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:31.857859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 875/1753] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:31 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:31.859172 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e804dabc-8c14-4ee7-aadd-175fcf75ba10 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.859451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e804dabc-8c14-4ee7-aadd-175fcf75ba10 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.859720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e804dabc-8c14-4ee7-aadd-175fcf75ba10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.892254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e804dabc-8c14-4ee7-aadd-175fcf75ba10 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:31.893792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e804dabc-8c14-4ee7-aadd-175fcf75ba10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:31.904988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 876/1754] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:31.910675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e804dabc-8c14-4ee7-aadd-175fcf75ba10 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:31.914603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:31.923826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e804dabc-8c14-4ee7-aadd-175fcf75ba10 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:31.928109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:31.930474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.930914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.931118 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.932074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:31.932341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:31.938363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:31.952848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 877/1755] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:31.953580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6167034a-ea22-4743-b468-21157342692e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:31.955425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:31.959093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd6e7c9-a6c6-42b0-8bd1-07f9b19baef2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:31.959906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd6e7c9-a6c6-42b0-8bd1-07f9b19baef2 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.960186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd6e7c9-a6c6-42b0-8bd1-07f9b19baef2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.960556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd6e7c9-a6c6-42b0-8bd1-07f9b19baef2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.968887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2427c-375a-49f3-9cb2-4c808de63085 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:31.971379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd6e7c9-a6c6-42b0-8bd1-07f9b19baef2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:31.973082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd6e7c9-a6c6-42b0-8bd1-07f9b19baef2 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:31.976054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1edf36-8639-4f4e-bcb7-669d3a022e4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:31.976753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1edf36-8639-4f4e-bcb7-669d3a022e4e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:31.977165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1edf36-8639-4f4e-bcb7-669d3a022e4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:31.977441 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1edf36-8639-4f4e-bcb7-669d3a022e4e admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:31.978080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1edf36-8639-4f4e-bcb7-669d3a022e4e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:31.981426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1edf36-8639-4f4e-bcb7-669d3a022e4e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:31.985651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 879/1756] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:32.024744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 878/1757] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:31 2022] POST /identity/v3/projects => generated 383 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:32.028928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd6e7c9-a6c6-42b0-8bd1-07f9b19baef2 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.039862 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd6e7c9-a6c6-42b0-8bd1-07f9b19baef2 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.128624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b30d76d-7289-4fd2-a03d-36aa91a9aaab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.129220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b30d76d-7289-4fd2-a03d-36aa91a9aaab admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.129504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b30d76d-7289-4fd2-a03d-36aa91a9aaab admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.129760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b30d76d-7289-4fd2-a03d-36aa91a9aaab admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.130390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b30d76d-7289-4fd2-a03d-36aa91a9aaab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.132785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b30d76d-7289-4fd2-a03d-36aa91a9aaab admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.160988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 880/1758] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:32 2022] POST /identity/v3/users => generated 292 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:32.165199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1edf36-8639-4f4e-bcb7-669d3a022e4e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.178676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1edf36-8639-4f4e-bcb7-669d3a022e4e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.190372 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358311ed-7435-414a-a96f-8c543db36617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.191824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358311ed-7435-414a-a96f-8c543db36617 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358311ed-7435-414a-a96f-8c543db36617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.192375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358311ed-7435-414a-a96f-8c543db36617 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.193558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358311ed-7435-414a-a96f-8c543db36617 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.195355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358311ed-7435-414a-a96f-8c543db36617 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.197666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:DELETE=7,keystone:SELECT=91,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:32.215517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b30d76d-7289-4fd2-a03d-36aa91a9aaab admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.216888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 879/1759] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:32.231794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b30d76d-7289-4fd2-a03d-36aa91a9aaab admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.240411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.241517 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.241977 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.242389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.243358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.245223 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.259907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 881/1760] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:32.264184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358311ed-7435-414a-a96f-8c543db36617 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.284571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358311ed-7435-414a-a96f-8c543db36617 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.294569 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c32151a-16f0-4dd0-bd1a-921ab8d740ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.295248 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c32151a-16f0-4dd0-bd1a-921ab8d740ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.295683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c32151a-16f0-4dd0-bd1a-921ab8d740ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.296018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c32151a-16f0-4dd0-bd1a-921ab8d740ea admin admin] PATH_INFO: `/v3/projects/9c080d1c9f0b45a9b6f6f98d538bb58e/users/6c3c44623fc04190b4bdf1d069eede82/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.320058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c32151a-16f0-4dd0-bd1a-921ab8d740ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c080d1c9f0b45a9b6f6f98d538bb58e, user_id=6c3c44623fc04190b4bdf1d069eede82, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.322835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c32151a-16f0-4dd0-bd1a-921ab8d740ea admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.338564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 880/1761] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:32 2022] PUT /identity/v3/projects/9c080d1c9f0b45a9b6f6f98d538bb58e/users/6c3c44623fc04190b4bdf1d069eede82/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:32.343266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.347115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:32.351758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5179f6fd-3528-4f2a-91d8-5121d757a9f2 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.352035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5179f6fd-3528-4f2a-91d8-5121d757a9f2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.352309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5179f6fd-3528-4f2a-91d8-5121d757a9f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.384970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db12f4-824f-4882-9b4a-309abafef25b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.388901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5179f6fd-3528-4f2a-91d8-5121d757a9f2 None None] MFA Rules not processed for user `6c3c44623fc04190b4bdf1d069eede82`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:32.390504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5179f6fd-3528-4f2a-91d8-5121d757a9f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:32.453961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.454988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.455379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.455658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] PATH_INFO: `/v3/projects/7786db617b14422da7b9e7117eeffdca` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.457314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=7786db617b14422da7b9e7117eeffdca)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.459254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.465277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 881/1762] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:32.477177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5179f6fd-3528-4f2a-91d8-5121d757a9f2 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.478616 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5179f6fd-3528-4f2a-91d8-5121d757a9f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:32.512223 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7786db617b14422da7b9e7117eeffdca', 'request_id': 'req-c507c997-6c90-461d-bd25-7ed98bd286eb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.516975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:32.520189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7786db617b14422da7b9e7117eeffdca', 'request_id': 'req-c507c997-6c90-461d-bd25-7ed98bd286eb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.528966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7786db617b14422da7b9e7117eeffdca', 'request_id': 'req-c507c997-6c90-461d-bd25-7ed98bd286eb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.541716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5179f6fd-3528-4f2a-91d8-5121d757a9f2 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.548851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] The token cache is being invalidate because project 7786db617b14422da7b9e7117eeffdca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:32.549156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.549857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c507c997-6c90-461d-bd25-7ed98bd286eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.553132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 882/1763] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:32 2022] DELETE /identity/v3/projects/7786db617b14422da7b9e7117eeffdca => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:32.559671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3b479-f630-4f13-825c-43d846f0753d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.560139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3b479-f630-4f13-825c-43d846f0753d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.560553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3b479-f630-4f13-825c-43d846f0753d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.593513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e3b3b479-f630-4f13-825c-43d846f0753d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:32.596145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e3b3b479-f630-4f13-825c-43d846f0753d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:32.661702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.663022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.663512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.664668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.666683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:32.680707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 883/1764] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:32.691078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a9abe-290a-4ec5-b0ba-b89991866f19 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.691405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a9abe-290a-4ec5-b0ba-b89991866f19 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.691777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a9abe-290a-4ec5-b0ba-b89991866f19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.713621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-564a9abe-290a-4ec5-b0ba-b89991866f19 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:32.714907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-564a9abe-290a-4ec5-b0ba-b89991866f19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:32.715782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:32.726694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 884/1765] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:32.731195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a9abe-290a-4ec5-b0ba-b89991866f19 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.754729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a9abe-290a-4ec5-b0ba-b89991866f19 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.759132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.760107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.760801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.760916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.761122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.761460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.762352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:32.762782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.764713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.772491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 882/1766] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:32 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:32.783223 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 885/1767] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:41:32.784377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.792704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.794421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:32.796086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5f90b1-1104-4cb6-9d77-c7c7327645da service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.800990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e14d3-6905-4250-90b3-9b4e22970954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.801594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e14d3-6905-4250-90b3-9b4e22970954 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.801908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e14d3-6905-4250-90b3-9b4e22970954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.802476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e14d3-6905-4250-90b3-9b4e22970954 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.803151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e14d3-6905-4250-90b3-9b4e22970954 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.805680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e14d3-6905-4250-90b3-9b4e22970954 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.807184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce1f4b8-0b9b-4565-a017-d1cadeba217d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.814106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.814841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.815119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.815390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] PATH_INFO: `/v3/users/6c3c44623fc04190b4bdf1d069eede82` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.816544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c3c44623fc04190b4bdf1d069eede82)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.818634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.835900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 883/1768] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:32 2022] POST /identity/v3/projects => generated 371 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:32.839951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e14d3-6905-4250-90b3-9b4e22970954 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.870929 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c3c44623fc04190b4bdf1d069eede82'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.878668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c3c44623fc04190b4bdf1d069eede82'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.879391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c3c44623fc04190b4bdf1d069eede82'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.880029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c3c44623fc04190b4bdf1d069eede82'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.888377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-28c46092-7897-4613-8819-84d313376eb8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c3c44623fc04190b4bdf1d069eede82'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:32.906737 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 886/1769] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:32 2022] DELETE /identity/v3/users/6c3c44623fc04190b4bdf1d069eede82 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:32.910309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e14d3-6905-4250-90b3-9b4e22970954 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:32.911730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190f9327-c89b-4495-b38f-0e3897944f7e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.912021 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190f9327-c89b-4495-b38f-0e3897944f7e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.912300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190f9327-c89b-4495-b38f-0e3897944f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.932998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-190f9327-c89b-4495-b38f-0e3897944f7e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:32.934218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-190f9327-c89b-4495-b38f-0e3897944f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:32.965461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:32.968230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:32.968844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:32.969126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:32.969915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:32.970838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:32.972742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:32.994398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 884/1770] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:32 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:32.998102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:32.999270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 887/1771] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:32.999723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.009153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190f9327-c89b-4495-b38f-0e3897944f7e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:33.010562 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-190f9327-c89b-4495-b38f-0e3897944f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.012429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088e3569-6994-4e0e-a72c-6602e36ae0d7 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:33.017461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139a63d0-d58e-476c-91fe-17205d000e18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:33.018070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a63d0-d58e-476c-91fe-17205d000e18 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.018359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a63d0-d58e-476c-91fe-17205d000e18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.018652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a63d0-d58e-476c-91fe-17205d000e18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.019292 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139a63d0-d58e-476c-91fe-17205d000e18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:33.020897 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139a63d0-d58e-476c-91fe-17205d000e18 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:33.036171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 885/1772] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:33.037333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139a63d0-d58e-476c-91fe-17205d000e18 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:33.055241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139a63d0-d58e-476c-91fe-17205d000e18 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:33.060443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11159d9c-627b-4b10-9648-cc07837922d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:33.061102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11159d9c-627b-4b10-9648-cc07837922d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.061335 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11159d9c-627b-4b10-9648-cc07837922d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.061612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11159d9c-627b-4b10-9648-cc07837922d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.062976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11159d9c-627b-4b10-9648-cc07837922d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:33.064978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11159d9c-627b-4b10-9648-cc07837922d0 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:33.082120 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 886/1773] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:33 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:33.082310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190f9327-c89b-4495-b38f-0e3897944f7e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:33.083384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11159d9c-627b-4b10-9648-cc07837922d0 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:33.084536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f736c0-4402-4d22-bacf-54618a972810 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:33.085227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f736c0-4402-4d22-bacf-54618a972810 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.085513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f736c0-4402-4d22-bacf-54618a972810 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.085786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f736c0-4402-4d22-bacf-54618a972810 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.095605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11159d9c-627b-4b10-9648-cc07837922d0 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:33.099612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f736c0-4402-4d22-bacf-54618a972810 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:33.101085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f736c0-4402-4d22-bacf-54618a972810 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:33.101342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:33.102155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.102437 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.103130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] PATH_INFO: `/v3/projects/f798b16ae6e4429c9c07793a0178f798/users/ed3d6b4184274791bce9c7d9b1ab028a/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.112774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 888/1774] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:33.124452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f798b16ae6e4429c9c07793a0178f798, user_id=ed3d6b4184274791bce9c7d9b1ab028a, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:33.128435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:33.139685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:33.149943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 887/1775] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:33 2022] PUT /identity/v3/projects/f798b16ae6e4429c9c07793a0178f798/users/ed3d6b4184274791bce9c7d9b1ab028a/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:33.151871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f52248-7e9f-47b2-8531-3ca0b2cce485 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.152156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f52248-7e9f-47b2-8531-3ca0b2cce485 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.152430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f52248-7e9f-47b2-8531-3ca0b2cce485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.174035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-78f52248-7e9f-47b2-8531-3ca0b2cce485 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:33.176370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-78f52248-7e9f-47b2-8531-3ca0b2cce485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.255532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 889/1776] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:33.265899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:33.269384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.367578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9650b8f8-e234-47e9-8b0e-8194bfe471e8 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:33.371894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:33.372687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.373109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.373522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.390466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:33.392207 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:33.404953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 888/1777] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:33.561519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f52248-7e9f-47b2-8531-3ca0b2cce485 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:33.571137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-78f52248-7e9f-47b2-8531-3ca0b2cce485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.590038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f52248-7e9f-47b2-8531-3ca0b2cce485 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:33.600235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:33.600895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.601164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.602112 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] PATH_INFO: `/v3/projects/9c080d1c9f0b45a9b6f6f98d538bb58e` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.604310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c080d1c9f0b45a9b6f6f98d538bb58e)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:33.609595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:33.661242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:33.663010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.678731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22564cb2-e5ef-41fa-a2db-c8c84232e4ca service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:33.687113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:33.687807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.689110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.689593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] PATH_INFO: `/v3/users/e0a9d27def4542ebb5b766b6ba9145bd` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.693004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0a9d27def4542ebb5b766b6ba9145bd)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:33.695004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:33.699813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c080d1c9f0b45a9b6f6f98d538bb58e', 'request_id': 'req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.708483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c080d1c9f0b45a9b6f6f98d538bb58e', 'request_id': 'req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.725116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c080d1c9f0b45a9b6f6f98d538bb58e', 'request_id': 'req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.773465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] The token cache is being invalidate because project 9c080d1c9f0b45a9b6f6f98d538bb58e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:33.775004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.776597 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e9a4032e-e2e8-4d23-b458-fa58bc3917b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.779141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0a9d27def4542ebb5b766b6ba9145bd'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.781385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 890/1778] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:33 2022] DELETE /identity/v3/projects/9c080d1c9f0b45a9b6f6f98d538bb58e => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:33.788200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6af10a-e958-4fe3-b4fa-78dca1d5b556 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.788738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6af10a-e958-4fe3-b4fa-78dca1d5b556 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.789555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6af10a-e958-4fe3-b4fa-78dca1d5b556 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.798312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0a9d27def4542ebb5b766b6ba9145bd'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.799052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0a9d27def4542ebb5b766b6ba9145bd'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.799752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0a9d27def4542ebb5b766b6ba9145bd'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.801992 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:33.812404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9bf17ddb-b84a-40d7-acdc-6f32c0b9754d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0a9d27def4542ebb5b766b6ba9145bd'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:33.847976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-fc6af10a-e958-4fe3-b4fa-78dca1d5b556 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:33.848123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 889/1779] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:33 2022] DELETE /identity/v3/users/e0a9d27def4542ebb5b766b6ba9145bd => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:33.850257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fc6af10a-e958-4fe3-b4fa-78dca1d5b556 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.853373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618cce4-e751-42f1-9d9e-89ccf8cfbf70 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.853805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618cce4-e751-42f1-9d9e-89ccf8cfbf70 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.854085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618cce4-e751-42f1-9d9e-89ccf8cfbf70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.888074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7618cce4-e751-42f1-9d9e-89ccf8cfbf70 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:33.889329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7618cce4-e751-42f1-9d9e-89ccf8cfbf70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.946538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 891/1780] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:33.950472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac7ef0-eabd-447f-bceb-92ee3975ecc5 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:33.951050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac7ef0-eabd-447f-bceb-92ee3975ecc5 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:33.951842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac7ef0-eabd-447f-bceb-92ee3975ecc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:33.964896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 890/1781] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:33.982127 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7618cce4-e751-42f1-9d9e-89ccf8cfbf70 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:33.984575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7618cce4-e751-42f1-9d9e-89ccf8cfbf70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:33.994672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-2cac7ef0-eabd-447f-bceb-92ee3975ecc5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:33.996165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2cac7ef0-eabd-447f-bceb-92ee3975ecc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:34.012869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 892/1782] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:34.013897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac7ef0-eabd-447f-bceb-92ee3975ecc5 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:34.014216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:34.025587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac7ef0-eabd-447f-bceb-92ee3975ecc5 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:34.032558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:34.033758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.034207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.034686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.035552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:34.035996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:34.039396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:34.058868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 893/1783] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:34.065413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:34.080206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7618cce4-e751-42f1-9d9e-89ccf8cfbf70 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:34.087631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d739525d-47bf-44f5-bc14-6ff91dd82f9f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:34.096823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cf8bf1-dc14-4121-b376-a2606959c305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:34.097898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cf8bf1-dc14-4121-b376-a2606959c305 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.098419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cf8bf1-dc14-4121-b376-a2606959c305 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.098774 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2147c0a-40ab-4bfe-99ba-3501aad256d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:34.099473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cf8bf1-dc14-4121-b376-a2606959c305 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.100116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2147c0a-40ab-4bfe-99ba-3501aad256d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.100816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cf8bf1-dc14-4121-b376-a2606959c305 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:34.101438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2147c0a-40ab-4bfe-99ba-3501aad256d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.101984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2147c0a-40ab-4bfe-99ba-3501aad256d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.103424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cf8bf1-dc14-4121-b376-a2606959c305 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:34.133416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2147c0a-40ab-4bfe-99ba-3501aad256d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:34.135369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2147c0a-40ab-4bfe-99ba-3501aad256d8 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:34.147315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 894/1784] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:34 2022] POST /identity/v3/projects => generated 373 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:34.153390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cf8bf1-dc14-4121-b376-a2606959c305 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:34.156065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 891/1785] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:34.236235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cf8bf1-dc14-4121-b376-a2606959c305 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:34.241926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b37261-c591-47ff-9aa1-0c888f5962fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:34.242604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b37261-c591-47ff-9aa1-0c888f5962fc admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.242946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b37261-c591-47ff-9aa1-0c888f5962fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.243287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b37261-c591-47ff-9aa1-0c888f5962fc admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.243997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b37261-c591-47ff-9aa1-0c888f5962fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:34.246065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b37261-c591-47ff-9aa1-0c888f5962fc admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:34.283949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 895/1786] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:34 2022] POST /identity/v3/users => generated 287 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:34.299518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2147c0a-40ab-4bfe-99ba-3501aad256d8 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:34.316645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2147c0a-40ab-4bfe-99ba-3501aad256d8 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:34.323884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e966fe3f-25ff-49a0-9ccf-40909c622586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:34.324628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966fe3f-25ff-49a0-9ccf-40909c622586 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.324922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966fe3f-25ff-49a0-9ccf-40909c622586 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.325324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966fe3f-25ff-49a0-9ccf-40909c622586 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.326138 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e966fe3f-25ff-49a0-9ccf-40909c622586 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:34.327949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e966fe3f-25ff-49a0-9ccf-40909c622586 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:34.355633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 892/1787] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:34 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:34.364095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b37261-c591-47ff-9aa1-0c888f5962fc admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:34.386369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b37261-c591-47ff-9aa1-0c888f5962fc admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:34.394303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211136a4-a3bb-44f5-9b37-c1c484f5684a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:34.395213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211136a4-a3bb-44f5-9b37-c1c484f5684a admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.395356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211136a4-a3bb-44f5-9b37-c1c484f5684a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.395769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211136a4-a3bb-44f5-9b37-c1c484f5684a admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.396577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211136a4-a3bb-44f5-9b37-c1c484f5684a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:34.398411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211136a4-a3bb-44f5-9b37-c1c484f5684a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:34.411430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 896/1788] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:34 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:34.417598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e966fe3f-25ff-49a0-9ccf-40909c622586 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:34.430981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e966fe3f-25ff-49a0-9ccf-40909c622586 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:34.440539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:34.441144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.441435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.441715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] PATH_INFO: `/v3/projects/bf8c75841cc14ea2b83df78ef60664e1/users/a0eda04ac81e4d0f8f3dd3abf17b9dc8/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.466363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf8c75841cc14ea2b83df78ef60664e1, user_id=a0eda04ac81e4d0f8f3dd3abf17b9dc8, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:34.469756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:34.504966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 893/1789] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:34 2022] PUT /identity/v3/projects/bf8c75841cc14ea2b83df78ef60664e1/users/a0eda04ac81e4d0f8f3dd3abf17b9dc8/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:34.516971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f18e667-0a6e-4e16-8125-ff0cdbeea9a9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.517657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f18e667-0a6e-4e16-8125-ff0cdbeea9a9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.518800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f18e667-0a6e-4e16-8125-ff0cdbeea9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.583603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1f18e667-0a6e-4e16-8125-ff0cdbeea9a9 None None] MFA Rules not processed for user `a0eda04ac81e4d0f8f3dd3abf17b9dc8`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:34.585009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1f18e667-0a6e-4e16-8125-ff0cdbeea9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:34.634317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:34.748404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 897/1790] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:34.763659 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:34.770249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:34.912342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5a24c9-d07e-4885-82ab-cc1e2694f4fa admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:34.923236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d7b0ca-56d7-4e2a-8c6a-6e80f92561db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:34.930344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d7b0ca-56d7-4e2a-8c6a-6e80f92561db service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:34.930758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d7b0ca-56d7-4e2a-8c6a-6e80f92561db service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:34.931003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d7b0ca-56d7-4e2a-8c6a-6e80f92561db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:34.958824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d7b0ca-56d7-4e2a-8c6a-6e80f92561db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:34.964792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d7b0ca-56d7-4e2a-8c6a-6e80f92561db service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:34.966851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f18e667-0a6e-4e16-8125-ff0cdbeea9a9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:34.970919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1f18e667-0a6e-4e16-8125-ff0cdbeea9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:34.990050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 894/1791] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:34 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:35.020173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d7b0ca-56d7-4e2a-8c6a-6e80f92561db service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:35.154397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d7b0ca-56d7-4e2a-8c6a-6e80f92561db service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:35.181505 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:35.182175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:35.182453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:35.183378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f18e667-0a6e-4e16-8125-ff0cdbeea9a9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:35.185631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] PATH_INFO: `/v3/users/a0eda04ac81e4d0f8f3dd3abf17b9dc8` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:35.186539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0eda04ac81e4d0f8f3dd3abf17b9dc8)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:35.189003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:35.191075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:35.191749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:35.192027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:35.192304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] PATH_INFO: `/v3/projects/88a8af88dac0499fa667717053378616` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:35.193458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=88a8af88dac0499fa667717053378616)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:35.195672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:35.244713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:35.281029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88a8af88dac0499fa667717053378616', 'request_id': 'req-f215f3c7-92e5-403e-80e6-d072e86730fc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.292343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0eda04ac81e4d0f8f3dd3abf17b9dc8'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.293347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88a8af88dac0499fa667717053378616', 'request_id': 'req-f215f3c7-92e5-403e-80e6-d072e86730fc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.316216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0eda04ac81e4d0f8f3dd3abf17b9dc8'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.317419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0eda04ac81e4d0f8f3dd3abf17b9dc8'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.318466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0eda04ac81e4d0f8f3dd3abf17b9dc8'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.320106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88a8af88dac0499fa667717053378616', 'request_id': 'req-f215f3c7-92e5-403e-80e6-d072e86730fc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.331175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-79062ffc-3d01-466e-87cd-911d2869f18a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0eda04ac81e4d0f8f3dd3abf17b9dc8'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.350275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] The token cache is being invalidate because project 88a8af88dac0499fa667717053378616 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:35.350604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.353738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.357338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 898/1792] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:34 2022] DELETE /identity/v3/projects/88a8af88dac0499fa667717053378616 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:35.361823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:35.365695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:35.370002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 895/1793] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:35 2022] DELETE /identity/v3/users/a0eda04ac81e4d0f8f3dd3abf17b9dc8 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:35.375081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2eea0-f4ef-4ebc-8798-b746b1fcc642 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:35.376740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2eea0-f4ef-4ebc-8798-b746b1fcc642 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:35.376740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad2eea0-f4ef-4ebc-8798-b746b1fcc642 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:35.397805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8ad2eea0-f4ef-4ebc-8798-b746b1fcc642 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:35.399445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8ad2eea0-f4ef-4ebc-8798-b746b1fcc642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:35.495635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 896/1794] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:35.496969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f215f3c7-92e5-403e-80e6-d072e86730fc admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:35.502789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:35.503426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad2eea0-f4ef-4ebc-8798-b746b1fcc642 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:35.503557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:35.503717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:35.503999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] PATH_INFO: `/v3/users/bbb565c99521478dbdb5615b7bf9d280` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:35.505447 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbb565c99521478dbdb5615b7bf9d280)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:35.506353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8ad2eea0-f4ef-4ebc-8798-b746b1fcc642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:35.507766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:35.509822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:35.592896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad2eea0-f4ef-4ebc-8798-b746b1fcc642 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:35.597975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:35.598853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:35.599135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:35.599914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:35.613451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbb565c99521478dbdb5615b7bf9d280'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.619834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:35.621295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:35.628444 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbb565c99521478dbdb5615b7bf9d280'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.631122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbb565c99521478dbdb5615b7bf9d280'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.632594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbb565c99521478dbdb5615b7bf9d280'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.635618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 897/1795] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:35.644809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbb565c99521478dbdb5615b7bf9d280'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:35.673223 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 899/1796] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:35 2022] DELETE /identity/v3/users/bbb565c99521478dbdb5615b7bf9d280 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:36.141988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:36.144151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:36.161171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7397dd74-9a5f-4bdc-aa64-6c38866dfc1e service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:36.239793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:36.245525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:36.345778 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:36.379825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:36.380930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:36.381219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:36.381495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] PATH_INFO: `/v3/projects/bf8c75841cc14ea2b83df78ef60664e1` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:36.383509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf8c75841cc14ea2b83df78ef60664e1)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:36.386334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:36.427258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a441775-0975-4993-881e-e7585213f7e0 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:36.433693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:36.434583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:36.434893 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:36.435357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] PATH_INFO: `/v3/projects/937f86c5cbb0457f905b637785c31f28` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:36.436761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=937f86c5cbb0457f905b637785c31f28)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:36.441319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:36.458825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf8c75841cc14ea2b83df78ef60664e1', 'request_id': 'req-717bcd49-c029-45dd-adec-af661cbb3372'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.465983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf8c75841cc14ea2b83df78ef60664e1', 'request_id': 'req-717bcd49-c029-45dd-adec-af661cbb3372'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.476055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf8c75841cc14ea2b83df78ef60664e1', 'request_id': 'req-717bcd49-c029-45dd-adec-af661cbb3372'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.496121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] The token cache is being invalidate because project bf8c75841cc14ea2b83df78ef60664e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:36.496383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.498918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.501447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 898/1797] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:36 2022] DELETE /identity/v3/projects/bf8c75841cc14ea2b83df78ef60664e1 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:36.502713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:36.504468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:36.548953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '937f86c5cbb0457f905b637785c31f28', 'request_id': 'req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.565078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '937f86c5cbb0457f905b637785c31f28', 'request_id': 'req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.583921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '937f86c5cbb0457f905b637785c31f28', 'request_id': 'req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.602208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] The token cache is being invalidate because project 937f86c5cbb0457f905b637785c31f28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:36.602540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.603343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c68a6c41-e405-4ee4-9b65-ac4b4fe4d0a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.609294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 900/1798] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:36 2022] DELETE /identity/v3/projects/937f86c5cbb0457f905b637785c31f28 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:36.611554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5713e6-f340-4886-85ae-09e9bae3cff9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:36.611931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5713e6-f340-4886-85ae-09e9bae3cff9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:36.612426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5713e6-f340-4886-85ae-09e9bae3cff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:36.614230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717bcd49-c029-45dd-adec-af661cbb3372 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:36.658079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-fb5713e6-f340-4886-85ae-09e9bae3cff9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:36.660289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fb5713e6-f340-4886-85ae-09e9bae3cff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:36.681947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:36.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:36.703977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:36.704341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:36.704991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] PATH_INFO: `/v3/users/e0336be78e4a45209b090cd2fe056915` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:36.706200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0336be78e4a45209b090cd2fe056915)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:36.708294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:36.752572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:36.778032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 901/1799] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:36.787909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385439d1-04e2-4b1a-9c27-3017a3bf3b86 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:36.788201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385439d1-04e2-4b1a-9c27-3017a3bf3b86 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:36.788479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385439d1-04e2-4b1a-9c27-3017a3bf3b86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:36.794038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0336be78e4a45209b090cd2fe056915'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.808805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0336be78e4a45209b090cd2fe056915'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.810032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0336be78e4a45209b090cd2fe056915'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.811263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0336be78e4a45209b090cd2fe056915'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.823385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-385439d1-04e2-4b1a-9c27-3017a3bf3b86 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:36.824970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-385439d1-04e2-4b1a-9c27-3017a3bf3b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:36.826732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-86bdb358-89fa-4254-8ba5-5f864c030bc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0336be78e4a45209b090cd2fe056915'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:36.837141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 902/1800] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:36.842732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34c493c-86bf-41a1-9710-3bad522e0ca2 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:36.842732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34c493c-86bf-41a1-9710-3bad522e0ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:36.842732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34c493c-86bf-41a1-9710-3bad522e0ca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:36.868744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f34c493c-86bf-41a1-9710-3bad522e0ca2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:36.869866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 899/1801] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:36 2022] DELETE /identity/v3/users/e0336be78e4a45209b090cd2fe056915 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:36.871927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f34c493c-86bf-41a1-9710-3bad522e0ca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:36.873142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2218b9-cf06-4587-831e-ff35ab8dbd3c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:36.873736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2218b9-cf06-4587-831e-ff35ab8dbd3c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:36.877816 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2218b9-cf06-4587-831e-ff35ab8dbd3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:36.909920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-be2218b9-cf06-4587-831e-ff35ab8dbd3c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:36.911493 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-be2218b9-cf06-4587-831e-ff35ab8dbd3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:36.957420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 903/1802] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:36.959053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea30b21-b130-43f4-99ff-860d9185fa6b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:36.959720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea30b21-b130-43f4-99ff-860d9185fa6b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:36.960000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea30b21-b130-43f4-99ff-860d9185fa6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:36.984072 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1ea30b21-b130-43f4-99ff-860d9185fa6b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:36.986702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1ea30b21-b130-43f4-99ff-860d9185fa6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:36.996919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 904/1803] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:36.997692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea30b21-b130-43f4-99ff-860d9185fa6b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.003624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 900/1804] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:37.008595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2218b9-cf06-4587-831e-ff35ab8dbd3c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.017483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea30b21-b130-43f4-99ff-860d9185fa6b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.023776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2218b9-cf06-4587-831e-ff35ab8dbd3c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.026925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.027560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.027839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.028109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.028760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:37.029061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.029209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.029933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.030276 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.030651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.031127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.031414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:37.031888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.033888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.051379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 905/1805] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:37.052370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.054796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:37.059420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 901/1806] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:41:37.060722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.078418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314a0cdb-df8f-429e-ac9f-aa1bd4aab8ad admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.087469 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e815a-715d-400f-8867-6df8f96b9a7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.088316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e815a-715d-400f-8867-6df8f96b9a7f admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.088673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e815a-715d-400f-8867-6df8f96b9a7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.089016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e815a-715d-400f-8867-6df8f96b9a7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.089735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50e815a-715d-400f-8867-6df8f96b9a7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.092589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50e815a-715d-400f-8867-6df8f96b9a7f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.134771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 902/1807] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:37 2022] POST /identity/v3/projects => generated 383 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:37.135982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e815a-715d-400f-8867-6df8f96b9a7f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.145953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295e96b1-8820-476c-8196-cf0576b0086b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.226954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:37.254980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe4ec4-1098-42e1-86f7-1ea1e3343f94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.255901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe4ec4-1098-42e1-86f7-1ea1e3343f94 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.256672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe4ec4-1098-42e1-86f7-1ea1e3343f94 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.257700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe4ec4-1098-42e1-86f7-1ea1e3343f94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.290377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e815a-715d-400f-8867-6df8f96b9a7f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.306916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd3ee04-c093-442b-b19b-1df742fcc32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.308159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd3ee04-c093-442b-b19b-1df742fcc32a admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.308587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd3ee04-c093-442b-b19b-1df742fcc32a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.309119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd3ee04-c093-442b-b19b-1df742fcc32a admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.309929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd3ee04-c093-442b-b19b-1df742fcc32a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.318169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd3ee04-c093-442b-b19b-1df742fcc32a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.340829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:37.364444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 903/1808] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:37 2022] POST /identity/v3/projects => generated 375 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:37.368281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd3ee04-c093-442b-b19b-1df742fcc32a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.369153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debe4ec4-1098-42e1-86f7-1ea1e3343f94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.370980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debe4ec4-1098-42e1-86f7-1ea1e3343f94 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.382507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 906/1809] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:37.383330 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe4ec4-1098-42e1-86f7-1ea1e3343f94 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.395584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe4ec4-1098-42e1-86f7-1ea1e3343f94 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.515234 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd3ee04-c093-442b-b19b-1df742fcc32a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.529325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639b85b1-e9d9-434e-9c41-a695ec19005e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.530347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b85b1-e9d9-434e-9c41-a695ec19005e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.530588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b85b1-e9d9-434e-9c41-a695ec19005e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.532540 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b85b1-e9d9-434e-9c41-a695ec19005e admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.534174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639b85b1-e9d9-434e-9c41-a695ec19005e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.537635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639b85b1-e9d9-434e-9c41-a695ec19005e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.591807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0cc54f-70e4-4ede-afd9-f1dd850d507d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.593408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cc54f-70e4-4ede-afd9-f1dd850d507d admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.595882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cc54f-70e4-4ede-afd9-f1dd850d507d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.596409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cc54f-70e4-4ede-afd9-f1dd850d507d admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.598153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0cc54f-70e4-4ede-afd9-f1dd850d507d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.604187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0cc54f-70e4-4ede-afd9-f1dd850d507d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.627093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 904/1810] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:37 2022] POST /identity/v3/users => generated 292 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:37.631391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639b85b1-e9d9-434e-9c41-a695ec19005e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.652104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 907/1811] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:37 2022] POST /identity/v3/users => generated 288 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:37.654804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639b85b1-e9d9-434e-9c41-a695ec19005e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.655958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0cc54f-70e4-4ede-afd9-f1dd850d507d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.667895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e34470-59af-4f7c-a8fe-3f9af6ebd5aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.669217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e34470-59af-4f7c-a8fe-3f9af6ebd5aa admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.669524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e34470-59af-4f7c-a8fe-3f9af6ebd5aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.669884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e34470-59af-4f7c-a8fe-3f9af6ebd5aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.672749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e34470-59af-4f7c-a8fe-3f9af6ebd5aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.674460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e34470-59af-4f7c-a8fe-3f9af6ebd5aa admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.681761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0cc54f-70e4-4ede-afd9-f1dd850d507d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.691906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b40c73c-71e3-42b0-b7f2-16fd7b37dfb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.692555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b40c73c-71e3-42b0-b7f2-16fd7b37dfb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.693277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b40c73c-71e3-42b0-b7f2-16fd7b37dfb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.693277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b40c73c-71e3-42b0-b7f2-16fd7b37dfb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.693751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b40c73c-71e3-42b0-b7f2-16fd7b37dfb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.700294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b40c73c-71e3-42b0-b7f2-16fd7b37dfb7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.707101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 905/1812] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:37.712149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e34470-59af-4f7c-a8fe-3f9af6ebd5aa admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.727531 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 908/1813] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:37.729494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b40c73c-71e3-42b0-b7f2-16fd7b37dfb7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.731090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e34470-59af-4f7c-a8fe-3f9af6ebd5aa admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.743260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8351fb-69ba-4765-b3aa-470220a07d91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.744755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8351fb-69ba-4765-b3aa-470220a07d91 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.745240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8351fb-69ba-4765-b3aa-470220a07d91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.746193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8351fb-69ba-4765-b3aa-470220a07d91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.747282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8351fb-69ba-4765-b3aa-470220a07d91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.750701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8351fb-69ba-4765-b3aa-470220a07d91 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.757349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b40c73c-71e3-42b0-b7f2-16fd7b37dfb7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.770643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 906/1814] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:37.772077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8351fb-69ba-4765-b3aa-470220a07d91 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.781457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040cd04a-6b20-42bf-ad23-4704e894304b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.782235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040cd04a-6b20-42bf-ad23-4704e894304b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.785598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8351fb-69ba-4765-b3aa-470220a07d91 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.787133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040cd04a-6b20-42bf-ad23-4704e894304b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.787512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040cd04a-6b20-42bf-ad23-4704e894304b admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.788455 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040cd04a-6b20-42bf-ad23-4704e894304b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.790095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040cd04a-6b20-42bf-ad23-4704e894304b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.798729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9199329-d706-45a9-b8cb-ec1d7eb5aa35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.801471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9199329-d706-45a9-b8cb-ec1d7eb5aa35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.801979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9199329-d706-45a9-b8cb-ec1d7eb5aa35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.802464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9199329-d706-45a9-b8cb-ec1d7eb5aa35 admin admin] PATH_INFO: `/v3/projects/2029d68d0ea74443964443dfb6e07aaa/users/737e742aabb34f32b1f6591a2841e107/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.822926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9199329-d706-45a9-b8cb-ec1d7eb5aa35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2029d68d0ea74443964443dfb6e07aaa, user_id=737e742aabb34f32b1f6591a2841e107, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:37.826280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9199329-d706-45a9-b8cb-ec1d7eb5aa35 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:37.835359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 909/1815] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:37 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:37.836446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040cd04a-6b20-42bf-ad23-4704e894304b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:37.857147 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040cd04a-6b20-42bf-ad23-4704e894304b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:37.882062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 907/1816] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:37 2022] PUT /identity/v3/projects/2029d68d0ea74443964443dfb6e07aaa/users/737e742aabb34f32b1f6591a2841e107/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:37.885002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1ed21-5c6d-4486-b9d3-fc5165f6c50e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.885291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1ed21-5c6d-4486-b9d3-fc5165f6c50e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.885563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1ed21-5c6d-4486-b9d3-fc5165f6c50e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:37.929860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-42a1ed21-5c6d-4486-b9d3-fc5165f6c50e None None] MFA Rules not processed for user `737e742aabb34f32b1f6591a2841e107`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:37.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-42a1ed21-5c6d-4486-b9d3-fc5165f6c50e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:37.978708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:37.995477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28945705-8955-432d-bc8f-2e915ed223b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:37.996166 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28945705-8955-432d-bc8f-2e915ed223b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:37.997380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28945705-8955-432d-bc8f-2e915ed223b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:37.997672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28945705-8955-432d-bc8f-2e915ed223b3 admin admin] PATH_INFO: `/v3/projects/8653edd746b44c1eb38d97d3c138933d/users/830dcc8551dc4b3cba5af5569253a1fb/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:38.037445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28945705-8955-432d-bc8f-2e915ed223b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8653edd746b44c1eb38d97d3c138933d, user_id=830dcc8551dc4b3cba5af5569253a1fb, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:38.041133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28945705-8955-432d-bc8f-2e915ed223b3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:38.079564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 910/1817] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:37 2022] PUT /identity/v3/projects/8653edd746b44c1eb38d97d3c138933d/users/830dcc8551dc4b3cba5af5569253a1fb/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:38.081280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22cfc27-44c4-40f2-938f-74eaa287dacb None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:38.081572 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22cfc27-44c4-40f2-938f-74eaa287dacb None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:38.081856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22cfc27-44c4-40f2-938f-74eaa287dacb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:38.099661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 908/1818] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:38.104975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1ed21-5c6d-4486-b9d3-fc5165f6c50e None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:38.123532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-d22cfc27-44c4-40f2-938f-74eaa287dacb None None] MFA Rules not processed for user `830dcc8551dc4b3cba5af5569253a1fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:38.125248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d22cfc27-44c4-40f2-938f-74eaa287dacb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:38.135547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1ed21-5c6d-4486-b9d3-fc5165f6c50e None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:38.198590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 911/1819] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:38 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:38.205591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:38.211474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22cfc27-44c4-40f2-938f-74eaa287dacb None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:38.217690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d22cfc27-44c4-40f2-938f-74eaa287dacb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:38.317919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde443c7-b606-47a7-8f03-57d0fd7a35d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:38.319367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde443c7-b606-47a7-8f03-57d0fd7a35d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:38.320978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde443c7-b606-47a7-8f03-57d0fd7a35d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:38.321602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde443c7-b606-47a7-8f03-57d0fd7a35d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:38.323566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde443c7-b606-47a7-8f03-57d0fd7a35d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:38.328752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde443c7-b606-47a7-8f03-57d0fd7a35d6 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:38.345742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22cfc27-44c4-40f2-938f-74eaa287dacb None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:38.354785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8b113d-2c4a-4422-9301-f9f116ad4297 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:38.355116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8b113d-2c4a-4422-9301-f9f116ad4297 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:38.355396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8b113d-2c4a-4422-9301-f9f116ad4297 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:38.355664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8b113d-2c4a-4422-9301-f9f116ad4297 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:38.378436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8b113d-2c4a-4422-9301-f9f116ad4297 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:38.380025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8b113d-2c4a-4422-9301-f9f116ad4297 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:38.380532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 909/1820] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:38 2022] POST /identity/v3/projects => generated 383 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:38.384651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde443c7-b606-47a7-8f03-57d0fd7a35d6 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:38.393725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 912/1821] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:38 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:38.537810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde443c7-b606-47a7-8f03-57d0fd7a35d6 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:38.544423 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3016a1-79de-48a6-bb2e-2ebc195af0ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:38.550305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3016a1-79de-48a6-bb2e-2ebc195af0ca admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:38.550852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3016a1-79de-48a6-bb2e-2ebc195af0ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:38.555316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3016a1-79de-48a6-bb2e-2ebc195af0ca admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:38.556307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3016a1-79de-48a6-bb2e-2ebc195af0ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:38.558361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3016a1-79de-48a6-bb2e-2ebc195af0ca admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:38.611728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 910/1822] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:38 2022] POST /identity/v3/users => generated 292 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:38.613696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8b113d-2c4a-4422-9301-f9f116ad4297 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:38.653294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8b113d-2c4a-4422-9301-f9f116ad4297 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:38.664954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfcda0c-b247-4d62-89b7-8d2fd92d7fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:38.665761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfcda0c-b247-4d62-89b7-8d2fd92d7fb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:38.666001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfcda0c-b247-4d62-89b7-8d2fd92d7fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:38.666419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfcda0c-b247-4d62-89b7-8d2fd92d7fb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:38.668320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfcda0c-b247-4d62-89b7-8d2fd92d7fb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:38.674988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfcda0c-b247-4d62-89b7-8d2fd92d7fb1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:38.715247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3016a1-79de-48a6-bb2e-2ebc195af0ca admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:38.715768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 913/1823] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:38 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:38.733772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3016a1-79de-48a6-bb2e-2ebc195af0ca admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:38.753758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:38.754410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:38.754741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:38.755091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:38.755872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:38.757603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:38.791720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 911/1824] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:38 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:38.795586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfcda0c-b247-4d62-89b7-8d2fd92d7fb1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:38.810251 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfcda0c-b247-4d62-89b7-8d2fd92d7fb1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:38.835425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28f4b65-389d-44bc-9638-49a5b2971b4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:38.836319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28f4b65-389d-44bc-9638-49a5b2971b4a admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:38.836620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28f4b65-389d-44bc-9638-49a5b2971b4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:38.836884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28f4b65-389d-44bc-9638-49a5b2971b4a admin admin] PATH_INFO: `/v3/projects/2a854d90b7214e6090921bb84aebf786/users/65cee7e355af4378ac1db4a1b946da8d/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:38.928179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28f4b65-389d-44bc-9638-49a5b2971b4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a854d90b7214e6090921bb84aebf786, user_id=65cee7e355af4378ac1db4a1b946da8d, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:38.937310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28f4b65-389d-44bc-9638-49a5b2971b4a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:38.969464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:38.974364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:39.008918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 914/1825] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:38 2022] PUT /identity/v3/projects/2a854d90b7214e6090921bb84aebf786/users/65cee7e355af4378ac1db4a1b946da8d/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:39.009908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57142e53-ce06-47c3-b180-501bcf63b761 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:39.012540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a947eb69-8254-45fc-b215-5c0bab2659b1 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:39.012685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a947eb69-8254-45fc-b215-5c0bab2659b1 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:39.012962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a947eb69-8254-45fc-b215-5c0bab2659b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:39.114669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a947eb69-8254-45fc-b215-5c0bab2659b1 None None] MFA Rules not processed for user `65cee7e355af4378ac1db4a1b946da8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:39.116281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a947eb69-8254-45fc-b215-5c0bab2659b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:39.154333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:39.188574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:39.189742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:39.191618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:39.191618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] PATH_INFO: `/v3/projects/e8ddf006ee284515ba78fa0280e2d498` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:39.198295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8ddf006ee284515ba78fa0280e2d498)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:39.205270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:39.253328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 915/1826] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:39.258601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a947eb69-8254-45fc-b215-5c0bab2659b1 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:39.292098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a947eb69-8254-45fc-b215-5c0bab2659b1 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:39.307939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:INSERT=4,keystone:DELETE=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:39.309219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c617476-fe2a-4384-87bf-2a9c09add04a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:39.310963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c617476-fe2a-4384-87bf-2a9c09add04a admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:39.312691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c617476-fe2a-4384-87bf-2a9c09add04a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:39.314146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c617476-fe2a-4384-87bf-2a9c09add04a admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:39.316336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c617476-fe2a-4384-87bf-2a9c09add04a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:39.320347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c617476-fe2a-4384-87bf-2a9c09add04a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:39.325307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8ddf006ee284515ba78fa0280e2d498', 'request_id': 'req-3e34b46d-cc49-48a9-a77a-51984f846100'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.332554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8ddf006ee284515ba78fa0280e2d498', 'request_id': 'req-3e34b46d-cc49-48a9-a77a-51984f846100'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.341547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8ddf006ee284515ba78fa0280e2d498', 'request_id': 'req-3e34b46d-cc49-48a9-a77a-51984f846100'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.358352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 916/1827] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:39 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:39.363823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c617476-fe2a-4384-87bf-2a9c09add04a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:39.367475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] The token cache is being invalidate because project e8ddf006ee284515ba78fa0280e2d498 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:39.367771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.372487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.381420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 912/1828] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:38 2022] DELETE /identity/v3/projects/e8ddf006ee284515ba78fa0280e2d498 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:39.383447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:39.384852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:39.470145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c617476-fe2a-4384-87bf-2a9c09add04a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:39.482275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:39.483210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:39.483489 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:39.483761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:39.484398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:39.486913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:39.537789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34b46d-cc49-48a9-a77a-51984f846100 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:39.544574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 917/1829] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:39 2022] POST /identity/v3/users => generated 299 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:39.549173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:39.557713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:39.596056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:39.596726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:39.597003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:39.597272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] PATH_INFO: `/v3/users/d46c96dc74ff4956bf4a1f83261bcaaa` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:39.600673 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d2f860-16fe-4c4c-9939-ebaae9c1f5f6 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:39.604301 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d46c96dc74ff4956bf4a1f83261bcaaa)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:39.609102 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:39.616096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e0a0c5-8b7e-43ae-b16f-c43ad74216f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:39.617140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0a0c5-8b7e-43ae-b16f-c43ad74216f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:39.617494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0a0c5-8b7e-43ae-b16f-c43ad74216f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:39.617765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0a0c5-8b7e-43ae-b16f-c43ad74216f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:39.618430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e0a0c5-8b7e-43ae-b16f-c43ad74216f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:39.620908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e0a0c5-8b7e-43ae-b16f-c43ad74216f3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:39.646865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 918/1830] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:39.649725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e0a0c5-8b7e-43ae-b16f-c43ad74216f3 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:39.684006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e0a0c5-8b7e-43ae-b16f-c43ad74216f3 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:39.714023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:39.717398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:39.718016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:39.718157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] PATH_INFO: `/v3/projects/b12eebcd55084dcf8b3f50df5f124cac/users/4ab22342a008422b95ed73d2e310660a/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:39.718927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd46c96dc74ff4956bf4a1f83261bcaaa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.737955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd46c96dc74ff4956bf4a1f83261bcaaa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd46c96dc74ff4956bf4a1f83261bcaaa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.740356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd46c96dc74ff4956bf4a1f83261bcaaa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.748163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b12eebcd55084dcf8b3f50df5f124cac, user_id=4ab22342a008422b95ed73d2e310660a, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:39.753583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:39.756004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd46c96dc74ff4956bf4a1f83261bcaaa'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:39.807352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 919/1831] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:39 2022] PUT /identity/v3/projects/b12eebcd55084dcf8b3f50df5f124cac/users/4ab22342a008422b95ed73d2e310660a/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:39.810915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:39.811358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 913/1832] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:39 2022] DELETE /identity/v3/users/d46c96dc74ff4956bf4a1f83261bcaaa => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:39.815654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:39.969935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad16340-733d-4be5-99f4-c8af7337ac2c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:39.977246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:40.092836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d3d0d4-a26c-4187-9c65-0d21faaf8428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:40.093700 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d3d0d4-a26c-4187-9c65-0d21faaf8428 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:40.094104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d3d0d4-a26c-4187-9c65-0d21faaf8428 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:40.094999 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d3d0d4-a26c-4187-9c65-0d21faaf8428 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:40.096023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d3d0d4-a26c-4187-9c65-0d21faaf8428 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:40.097912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d3d0d4-a26c-4187-9c65-0d21faaf8428 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:40.113438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 920/1833] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:39 2022] GET /identity/v3/roles => generated 1420 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:40.118227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:40.131144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b338dd-486d-433a-bfd3-da50fc2c57a2 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:40.136087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:40.136968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:40.137249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:40.137521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] PATH_INFO: `/v3/projects/b12eebcd55084dcf8b3f50df5f124cac/users/4ab22342a008422b95ed73d2e310660a/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:40.139662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b12eebcd55084dcf8b3f50df5f124cac, user_id=4ab22342a008422b95ed73d2e310660a, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:40.144064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:40.163938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 914/1834] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:40 2022] PUT /identity/v3/projects/b12eebcd55084dcf8b3f50df5f124cac/users/4ab22342a008422b95ed73d2e310660a/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:40.176537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3336c6d-34d4-4ce9-a781-5d9a6f42c774 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:40.176826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3336c6d-34d4-4ce9-a781-5d9a6f42c774 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:40.177104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3336c6d-34d4-4ce9-a781-5d9a6f42c774 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:40.248267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c3336c6d-34d4-4ce9-a781-5d9a6f42c774 None None] MFA Rules not processed for user `4ab22342a008422b95ed73d2e310660a`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:40.249749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c3336c6d-34d4-4ce9-a781-5d9a6f42c774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:40.330692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:40.333132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:40.416773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:DELETE=15,keystone:SELECT=82,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:40.433553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 921/1835] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:40 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:40.448522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3336c6d-34d4-4ce9-a781-5d9a6f42c774 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:40.450679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c3336c6d-34d4-4ce9-a781-5d9a6f42c774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:40.501346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d788f335-c614-469d-aaf4-f4f1fd177ef4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:40.508296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:40.509070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:40.509621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:40.509962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] PATH_INFO: `/v3/projects/c738162d881d4baa934897639f60a2d4` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:40.512139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c738162d881d4baa934897639f60a2d4)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:40.514996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:40.576591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3336c6d-34d4-4ce9-a781-5d9a6f42c774 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:40.616154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8a471b-1c41-4351-8946-0bf42f98106b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:40.617591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a471b-1c41-4351-8946-0bf42f98106b service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:40.618029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a471b-1c41-4351-8946-0bf42f98106b service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:40.618433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a471b-1c41-4351-8946-0bf42f98106b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:40.623482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1c8a471b-1c41-4351-8946-0bf42f98106b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:40.635738 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c738162d881d4baa934897639f60a2d4', 'request_id': 'req-55634a8b-6617-478d-bd9c-9707b3a85e53'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.643162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c738162d881d4baa934897639f60a2d4', 'request_id': 'req-55634a8b-6617-478d-bd9c-9707b3a85e53'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.652956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c738162d881d4baa934897639f60a2d4', 'request_id': 'req-55634a8b-6617-478d-bd9c-9707b3a85e53'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.671627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] The token cache is being invalidate because project c738162d881d4baa934897639f60a2d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:40.672025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.673344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.675290 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 915/1836] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:40 2022] DELETE /identity/v3/projects/c738162d881d4baa934897639f60a2d4 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:40.679460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:40.681255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:40.707675 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:40.733552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8a471b-1c41-4351-8946-0bf42f98106b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:40.735798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8a471b-1c41-4351-8946-0bf42f98106b service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:40.748365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 922/1837] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:40 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:40.783103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55634a8b-6617-478d-bd9c-9707b3a85e53 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:40.789228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:40.789933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:40.790218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:40.790518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] PATH_INFO: `/v3/users/2c8c63a0e580407f94977a3a0e6ca01f` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:40.791711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c8c63a0e580407f94977a3a0e6ca01f)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:40.793865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:40.905306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c8c63a0e580407f94977a3a0e6ca01f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.925653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c8c63a0e580407f94977a3a0e6ca01f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.927765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c8c63a0e580407f94977a3a0e6ca01f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.928366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c8c63a0e580407f94977a3a0e6ca01f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.937881 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:40.949272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c8c63a0e580407f94977a3a0e6ca01f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:40.987855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 916/1838] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:40 2022] DELETE /identity/v3/users/2c8c63a0e580407f94977a3a0e6ca01f => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:41.142317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b003f94e-99a6-4acb-aa7b-c0f95e82f0a3 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.142695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b003f94e-99a6-4acb-aa7b-c0f95e82f0a3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.143037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b003f94e-99a6-4acb-aa7b-c0f95e82f0a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.223459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b003f94e-99a6-4acb-aa7b-c0f95e82f0a3 None None] MFA Rules not processed for user `ed3d6b4184274791bce9c7d9b1ab028a`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:41.224726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b003f94e-99a6-4acb-aa7b-c0f95e82f0a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:41.441231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 923/1839] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:41 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:41.442851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:41.452226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:41.619276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b003f94e-99a6-4acb-aa7b-c0f95e82f0a3 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:41.620818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b003f94e-99a6-4acb-aa7b-c0f95e82f0a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:41.632425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44350e75-fff8-408a-8a5f-02d54127b0c7 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:41.640139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c33e281-dbca-4610-af4a-fcb82d0c83f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:41.641480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c33e281-dbca-4610-af4a-fcb82d0c83f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.641861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c33e281-dbca-4610-af4a-fcb82d0c83f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.642355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c33e281-dbca-4610-af4a-fcb82d0c83f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.642985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c33e281-dbca-4610-af4a-fcb82d0c83f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:41.645173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c33e281-dbca-4610-af4a-fcb82d0c83f4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:41.653087 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b003f94e-99a6-4acb-aa7b-c0f95e82f0a3 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:41.665860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:41.666692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.667089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.667375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] PATH_INFO: `/v3/projects/d6ceda4660fc49e3b8d9010acf282cdc` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.668678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6ceda4660fc49e3b8d9010acf282cdc)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:41.670857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:41.683825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 917/1840] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:41 2022] POST /identity/v3/projects => generated 371 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:41.684810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c33e281-dbca-4610-af4a-fcb82d0c83f4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:41.745348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6ceda4660fc49e3b8d9010acf282cdc', 'request_id': 'req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:41.751711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6ceda4660fc49e3b8d9010acf282cdc', 'request_id': 'req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:41.764912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6ceda4660fc49e3b8d9010acf282cdc', 'request_id': 'req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:41.766927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c33e281-dbca-4610-af4a-fcb82d0c83f4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:41.777889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] The token cache is being invalidate because project d6ceda4660fc49e3b8d9010acf282cdc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:41.778184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:41.779220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-fc1cc555-ecfa-48f9-9d35-199e0b6d31b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:41.780795 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 924/1841] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:41 2022] DELETE /identity/v3/projects/d6ceda4660fc49e3b8d9010acf282cdc => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:41.785801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72a92cc-8a11-40e4-892d-62a350652b2c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.786202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72a92cc-8a11-40e4-892d-62a350652b2c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.786515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72a92cc-8a11-40e4-892d-62a350652b2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.811604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a72a92cc-8a11-40e4-892d-62a350652b2c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:41.814412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a72a92cc-8a11-40e4-892d-62a350652b2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:41.836105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:41.836717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.837001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.837276 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.837911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:41.839858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:41.858691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:41.865657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 918/1842] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:41 2022] POST /identity/v3/users => generated 292 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:41.868883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:41.869811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:41.871776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:41.883775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 925/1843] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:41.884882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6314dc2-09d5-42e0-b227-6a9c14f1f3e7 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:41.892239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922c8591-d920-4e9c-9667-44dcad1df0ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:41.892402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea6a734-b24d-41a3-b9f7-9199c36f6d2b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.892648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea6a734-b24d-41a3-b9f7-9199c36f6d2b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.892878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c8591-d920-4e9c-9667-44dcad1df0ec admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.893001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea6a734-b24d-41a3-b9f7-9199c36f6d2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.893194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c8591-d920-4e9c-9667-44dcad1df0ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.893430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c8591-d920-4e9c-9667-44dcad1df0ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.894082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922c8591-d920-4e9c-9667-44dcad1df0ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:41.895736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922c8591-d920-4e9c-9667-44dcad1df0ec admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:41.910627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 919/1844] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:41 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:41.913330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922c8591-d920-4e9c-9667-44dcad1df0ec admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:41.921840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-eea6a734-b24d-41a3-b9f7-9199c36f6d2b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:41.923549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-eea6a734-b24d-41a3-b9f7-9199c36f6d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:41.930298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922c8591-d920-4e9c-9667-44dcad1df0ec admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:41.932402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 926/1845] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:41.936012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a24d901-5eb4-45cc-bc93-4016d751fd0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:41.936672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea6a734-b24d-41a3-b9f7-9199c36f6d2b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:41.937490 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a24d901-5eb4-45cc-bc93-4016d751fd0f admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.939534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a24d901-5eb4-45cc-bc93-4016d751fd0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.940208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a24d901-5eb4-45cc-bc93-4016d751fd0f admin admin] PATH_INFO: `/v3/projects/995ba646a2c047c5a9b1df24325ff98d/users/fb124845696e4ed6ad0b7c566f8bbd50/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.949117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea6a734-b24d-41a3-b9f7-9199c36f6d2b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:41.956980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:41.957677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:41.958030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:41.958572 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:41.959058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:41.959366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:41.962977 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:41.963231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a24d901-5eb4-45cc-bc93-4016d751fd0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=995ba646a2c047c5a9b1df24325ff98d, user_id=fb124845696e4ed6ad0b7c566f8bbd50, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:41.966167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a24d901-5eb4-45cc-bc93-4016d751fd0f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:41.979638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 927/1846] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:41:41.983615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 920/1847] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:41 2022] PUT /identity/v3/projects/995ba646a2c047c5a9b1df24325ff98d/users/fb124845696e4ed6ad0b7c566f8bbd50/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:41.983855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:41.987486 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a24d901-5eb4-45cc-bc93-4016d751fd0f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:41.994865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602da242-d211-4ed5-b825-e3ee8ca6d59a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:42.066727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a24d901-5eb4-45cc-bc93-4016d751fd0f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:42.087581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af19f37-d834-419a-a435-b81110ec272a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:42.088052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1189af3c-c384-4416-b8e2-b0ca7f9e9d6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:42.088201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af19f37-d834-419a-a435-b81110ec272a admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.088475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af19f37-d834-419a-a435-b81110ec272a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.088710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1189af3c-c384-4416-b8e2-b0ca7f9e9d6d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.088828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af19f37-d834-419a-a435-b81110ec272a admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.088992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1189af3c-c384-4416-b8e2-b0ca7f9e9d6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.089263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1189af3c-c384-4416-b8e2-b0ca7f9e9d6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.089365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af19f37-d834-419a-a435-b81110ec272a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:42.089919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1189af3c-c384-4416-b8e2-b0ca7f9e9d6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:42.091292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af19f37-d834-419a-a435-b81110ec272a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:42.091552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1189af3c-c384-4416-b8e2-b0ca7f9e9d6d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:42.104342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 921/1848] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:41 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:42.108473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1189af3c-c384-4416-b8e2-b0ca7f9e9d6d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:42.120832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1189af3c-c384-4416-b8e2-b0ca7f9e9d6d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:42.128971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decb9406-eac1-46b1-bb73-85d7f860310a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:42.129381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb9406-eac1-46b1-bb73-85d7f860310a admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.129754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb9406-eac1-46b1-bb73-85d7f860310a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.130104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb9406-eac1-46b1-bb73-85d7f860310a admin admin] PATH_INFO: `/v3/projects/995ba646a2c047c5a9b1df24325ff98d/users/fb124845696e4ed6ad0b7c566f8bbd50/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.133805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decb9406-eac1-46b1-bb73-85d7f860310a admin admin] RBAC: Authorizing `identity:create_grant(project_id=995ba646a2c047c5a9b1df24325ff98d, user_id=fb124845696e4ed6ad0b7c566f8bbd50, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:42.141839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 928/1849] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:41 2022] POST /identity/v3/projects => generated 363 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:42.142325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decb9406-eac1-46b1-bb73-85d7f860310a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:42.145997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af19f37-d834-419a-a435-b81110ec272a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:42.151989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 922/1850] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:42 2022] PUT /identity/v3/projects/995ba646a2c047c5a9b1df24325ff98d/users/fb124845696e4ed6ad0b7c566f8bbd50/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:42.237311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af19f37-d834-419a-a435-b81110ec272a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:42.305964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1b38cf-0f18-4139-a6ba-3c958329727f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:42.306638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1b38cf-0f18-4139-a6ba-3c958329727f admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.306753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1b38cf-0f18-4139-a6ba-3c958329727f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.307281 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1b38cf-0f18-4139-a6ba-3c958329727f admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.307921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1b38cf-0f18-4139-a6ba-3c958329727f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:42.309597 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1b38cf-0f18-4139-a6ba-3c958329727f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:42.332689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:42.342816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 929/1851] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:42 2022] POST /identity/v3/users => generated 282 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:42.348440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decb9406-eac1-46b1-bb73-85d7f860310a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:42.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decb9406-eac1-46b1-bb73-85d7f860310a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:42.369313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601eec4-7535-4916-90d3-4b24296be1cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:42.370077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601eec4-7535-4916-90d3-4b24296be1cf admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.370363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601eec4-7535-4916-90d3-4b24296be1cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.370668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601eec4-7535-4916-90d3-4b24296be1cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.371361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7601eec4-7535-4916-90d3-4b24296be1cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:42.372972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7601eec4-7535-4916-90d3-4b24296be1cf admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:42.387545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 923/1852] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:42 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:42.389224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1b38cf-0f18-4139-a6ba-3c958329727f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:42.412403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1b38cf-0f18-4139-a6ba-3c958329727f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:42.425536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974bd3f4-9956-433d-b1a9-f6ebcce122c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:42.426540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974bd3f4-9956-433d-b1a9-f6ebcce122c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.427059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974bd3f4-9956-433d-b1a9-f6ebcce122c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.427645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974bd3f4-9956-433d-b1a9-f6ebcce122c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.428477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974bd3f4-9956-433d-b1a9-f6ebcce122c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:42.430818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974bd3f4-9956-433d-b1a9-f6ebcce122c8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:42.450433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 930/1853] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:42 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:42.455303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601eec4-7535-4916-90d3-4b24296be1cf admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:42.466295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601eec4-7535-4916-90d3-4b24296be1cf admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:42.472069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:42.472746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.473020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.473293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] PATH_INFO: `/v3/projects/370471a6d2ef476db1f8dc94af08a88d/users/d185d73daa1a440088b4c6fa7134dce0/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.503110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=370471a6d2ef476db1f8dc94af08a88d, user_id=d185d73daa1a440088b4c6fa7134dce0, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:42.505827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:42.555112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 924/1854] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:42 2022] PUT /identity/v3/projects/370471a6d2ef476db1f8dc94af08a88d/users/d185d73daa1a440088b4c6fa7134dce0/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:42.573711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f44af-b9ee-4300-9b53-070f902c9d2c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.573711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f44af-b9ee-4300-9b53-070f902c9d2c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.573711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f44af-b9ee-4300-9b53-070f902c9d2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.618017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a92f44af-b9ee-4300-9b53-070f902c9d2c None None] MFA Rules not processed for user `d185d73daa1a440088b4c6fa7134dce0`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:42.619599 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a92f44af-b9ee-4300-9b53-070f902c9d2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:42.750667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 931/1855] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:42 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:42.762069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:42.763999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:42.846622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8e9ffc-b227-4a03-b7e8-d9da7972dfe4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:42.851191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3602ee99-c9cd-4aad-b9c9-a9879c2d2efb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:42.852484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3602ee99-c9cd-4aad-b9c9-a9879c2d2efb service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:42.852773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3602ee99-c9cd-4aad-b9c9-a9879c2d2efb service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:42.853053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3602ee99-c9cd-4aad-b9c9-a9879c2d2efb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:42.863941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3602ee99-c9cd-4aad-b9c9-a9879c2d2efb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:42.865457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3602ee99-c9cd-4aad-b9c9-a9879c2d2efb service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:42.877658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 925/1856] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:42 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:45.350835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92f44af-b9ee-4300-9b53-070f902c9d2c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:45.417269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92f44af-b9ee-4300-9b53-070f902c9d2c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:45.432097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:45.433180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:45.433901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:45.434168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:45.438370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:45.714257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:45.731174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:45.765481 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 932/1857] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:45 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:48.496673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01683bfa-982e-4c85-90d9-29215d28b961 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:48.497329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01683bfa-982e-4c85-90d9-29215d28b961 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:48.497329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01683bfa-982e-4c85-90d9-29215d28b961 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:48.563354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-01683bfa-982e-4c85-90d9-29215d28b961 None None] MFA Rules not processed for user `fb124845696e4ed6ad0b7c566f8bbd50`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:48.565110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-01683bfa-982e-4c85-90d9-29215d28b961 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:48.569946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:48.731886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 926/1858] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:48 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:48.747827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:48.750573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:48.930248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c0d0d0-6352-41da-aad9-94cfbbe7be8b service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:48.939354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01683bfa-982e-4c85-90d9-29215d28b961 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:48.948072 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b3a357-98d9-482d-8534-ac22972c942b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:48.948797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b3a357-98d9-482d-8534-ac22972c942b service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:48.949081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b3a357-98d9-482d-8534-ac22972c942b service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:48.949422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b3a357-98d9-482d-8534-ac22972c942b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:48.951526 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-51b3a357-98d9-482d-8534-ac22972c942b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:48.966212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:48.973789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01683bfa-982e-4c85-90d9-29215d28b961 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:48.991595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:48.992545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:48.992831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:48.993112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:48.996761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:49.051276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b3a357-98d9-482d-8534-ac22972c942b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:49.054980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b3a357-98d9-482d-8534-ac22972c942b service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:49.085336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 933/1859] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:48 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:49.193768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:49.206253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:49.242496 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 927/1860] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:48 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:51.126044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee52046-3020-4f34-bed8-78df98b377a7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:51.126577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee52046-3020-4f34-bed8-78df98b377a7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:51.126669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee52046-3020-4f34-bed8-78df98b377a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:51.151299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-aee52046-3020-4f34-bed8-78df98b377a7 None None] MFA Rules not processed for user `ed3d6b4184274791bce9c7d9b1ab028a`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:51.152658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-aee52046-3020-4f34-bed8-78df98b377a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:51.179428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 934/1861] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:51 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:52.002748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:52.004849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:52.106613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b824d5-fa6a-4de1-898e-e6bfbe8ec41c service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:52.119205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:52.122558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:52.123244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:52.123718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] PATH_INFO: `/v3/users/8118f0c698aa428a93cb054abac30c2e` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:52.125883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8118f0c698aa428a93cb054abac30c2e)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:52.129563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:52.276325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:52.294975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8118f0c698aa428a93cb054abac30c2e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.307780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8118f0c698aa428a93cb054abac30c2e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.307780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8118f0c698aa428a93cb054abac30c2e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.308745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8118f0c698aa428a93cb054abac30c2e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.323696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a71e8d2f-bc91-4333-b652-1635f1299338 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8118f0c698aa428a93cb054abac30c2e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.365119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 928/1862] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:52 2022] DELETE /identity/v3/users/8118f0c698aa428a93cb054abac30c2e => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:52.660887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee52046-3020-4f34-bed8-78df98b377a7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:52.677363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee52046-3020-4f34-bed8-78df98b377a7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:52.843197 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:52.844419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:52.844886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:52.845222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] PATH_INFO: `/v3/projects/f249d9041c6749f58973a5b0867b07ce` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:52.846930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f249d9041c6749f58973a5b0867b07ce)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:52.848907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:52.915317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f249d9041c6749f58973a5b0867b07ce', 'request_id': 'req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.920945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f249d9041c6749f58973a5b0867b07ce', 'request_id': 'req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.928963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f249d9041c6749f58973a5b0867b07ce', 'request_id': 'req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.940414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] The token cache is being invalidate because project f249d9041c6749f58973a5b0867b07ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:52.940703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.941336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2eeed5a7-8514-41f4-a40a-1c879c3a3e35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:52.943009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 935/1863] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:52 2022] DELETE /identity/v3/projects/f249d9041c6749f58973a5b0867b07ce => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:52.948071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed891fe-fb0d-41cb-a800-ea45c0431e5c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:52.948356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed891fe-fb0d-41cb-a800-ea45c0431e5c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:52.948628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed891fe-fb0d-41cb-a800-ea45c0431e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:52.970719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-aed891fe-fb0d-41cb-a800-ea45c0431e5c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:52.972074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-aed891fe-fb0d-41cb-a800-ea45c0431e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:53.044632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 929/1864] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:53.058609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4cd4a9-8712-41aa-9503-c2cf9db565b1 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.059129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4cd4a9-8712-41aa-9503-c2cf9db565b1 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.059548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4cd4a9-8712-41aa-9503-c2cf9db565b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.082702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-7d4cd4a9-8712-41aa-9503-c2cf9db565b1 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:53.084289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7d4cd4a9-8712-41aa-9503-c2cf9db565b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:53.093037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 936/1865] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:53.098297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed891fe-fb0d-41cb-a800-ea45c0431e5c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.113716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed891fe-fb0d-41cb-a800-ea45c0431e5c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:53.119042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:53.119693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.119974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.120246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.120939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:53.121240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:53.122940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:53.136275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 930/1866] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:53.141272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4cd4a9-8712-41aa-9503-c2cf9db565b1 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.150953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4cd4a9-8712-41aa-9503-c2cf9db565b1 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:53.158462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1220f1-49ea-474d-93c6-3854b4719538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:53.159337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1220f1-49ea-474d-93c6-3854b4719538 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.159725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1220f1-49ea-474d-93c6-3854b4719538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.160191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1220f1-49ea-474d-93c6-3854b4719538 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.160894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1220f1-49ea-474d-93c6-3854b4719538 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:53.162960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1220f1-49ea-474d-93c6-3854b4719538 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:53.198542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 937/1867] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:53 2022] POST /identity/v3/projects => generated 405 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:53.204956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.220178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df439b52-f95c-410a-8eec-1ffa53694d08 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:53.301542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10908b3-9089-4a66-9721-b4b69e6d9f9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:53.314512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10908b3-9089-4a66-9721-b4b69e6d9f9a admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.314512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10908b3-9089-4a66-9721-b4b69e6d9f9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.314512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10908b3-9089-4a66-9721-b4b69e6d9f9a admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.314512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10908b3-9089-4a66-9721-b4b69e6d9f9a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:53.314512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10908b3-9089-4a66-9721-b4b69e6d9f9a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:53.352929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 931/1868] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:53 2022] POST /identity/v3/users => generated 303 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:53.354904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1220f1-49ea-474d-93c6-3854b4719538 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.370348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1220f1-49ea-474d-93c6-3854b4719538 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:53.382074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1430f1-69e5-4f3d-8bf2-1189fa376a28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:53.382950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1430f1-69e5-4f3d-8bf2-1189fa376a28 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.383458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1430f1-69e5-4f3d-8bf2-1189fa376a28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.383948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1430f1-69e5-4f3d-8bf2-1189fa376a28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.385059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1430f1-69e5-4f3d-8bf2-1189fa376a28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:53.386930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1430f1-69e5-4f3d-8bf2-1189fa376a28 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:53.389182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:53.410241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 938/1869] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:53 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:53.411623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10908b3-9089-4a66-9721-b4b69e6d9f9a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.423362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10908b3-9089-4a66-9721-b4b69e6d9f9a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:53.428567 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f7c189-1179-4281-a8a8-5978092a4a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:53.429263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f7c189-1179-4281-a8a8-5978092a4a93 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.429643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f7c189-1179-4281-a8a8-5978092a4a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.430033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f7c189-1179-4281-a8a8-5978092a4a93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.431201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f7c189-1179-4281-a8a8-5978092a4a93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:53.432915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f7c189-1179-4281-a8a8-5978092a4a93 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:53.454095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 932/1870] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:53 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:53.458317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1430f1-69e5-4f3d-8bf2-1189fa376a28 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.473085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1430f1-69e5-4f3d-8bf2-1189fa376a28 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:53.478708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b811171e-1721-4794-b4c1-d0127cac9deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:53.486782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b811171e-1721-4794-b4c1-d0127cac9deb admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.489212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b811171e-1721-4794-b4c1-d0127cac9deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.490277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b811171e-1721-4794-b4c1-d0127cac9deb admin admin] PATH_INFO: `/v3/projects/c0368a9c77754721b730abf25d485814/users/2bb381599c3549e996ce8b27a4f42f69/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.516670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b811171e-1721-4794-b4c1-d0127cac9deb admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0368a9c77754721b730abf25d485814, user_id=2bb381599c3549e996ce8b27a4f42f69, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:53.520241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b811171e-1721-4794-b4c1-d0127cac9deb admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:53.545358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 939/1871] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:53 2022] PUT /identity/v3/projects/c0368a9c77754721b730abf25d485814/users/2bb381599c3549e996ce8b27a4f42f69/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:53.555914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a279747c-bc82-4a1f-8736-08a21356d1ab None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.556306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a279747c-bc82-4a1f-8736-08a21356d1ab None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.556650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a279747c-bc82-4a1f-8736-08a21356d1ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.615820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a279747c-bc82-4a1f-8736-08a21356d1ab None None] MFA Rules not processed for user `2bb381599c3549e996ce8b27a4f42f69`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:53.617689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a279747c-bc82-4a1f-8736-08a21356d1ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:53.628794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:53.767467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 933/1872] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:53 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:53.772253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b811171e-1721-4794-b4c1-d0127cac9deb admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.876842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b811171e-1721-4794-b4c1-d0127cac9deb admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:53.883821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d49e97-d528-4f57-945f-1f8254adce4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:53.884686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d49e97-d528-4f57-945f-1f8254adce4d admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:53.885128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d49e97-d528-4f57-945f-1f8254adce4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:53.885695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d49e97-d528-4f57-945f-1f8254adce4d admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:53.886540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d49e97-d528-4f57-945f-1f8254adce4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:53.889092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d49e97-d528-4f57-945f-1f8254adce4d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:53.925064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 940/1873] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:53 2022] POST /identity/v3/projects => generated 403 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:53.930410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d49e97-d528-4f57-945f-1f8254adce4d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.930584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a279747c-bc82-4a1f-8736-08a21356d1ab None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:53.932042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a279747c-bc82-4a1f-8736-08a21356d1ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:53.959742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a279747c-bc82-4a1f-8736-08a21356d1ab None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:54.071012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d49e97-d528-4f57-945f-1f8254adce4d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:54.079213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0ad68d-c0a6-46e4-a7f8-2b4931dbed42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:54.081468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ad68d-c0a6-46e4-a7f8-2b4931dbed42 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.082224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ad68d-c0a6-46e4-a7f8-2b4931dbed42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.082973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ad68d-c0a6-46e4-a7f8-2b4931dbed42 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.083952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0ad68d-c0a6-46e4-a7f8-2b4931dbed42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:54.086128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0ad68d-c0a6-46e4-a7f8-2b4931dbed42 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:54.097082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:54.097859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.098738 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.099181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] PATH_INFO: `/v3/users/737e742aabb34f32b1f6591a2841e107` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.100958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] RBAC: Authorizing `identity:delete_user(user_id=737e742aabb34f32b1f6591a2841e107)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:54.105790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:54.120133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 941/1874] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:53 2022] POST /identity/v3/users => generated 308 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:54.124318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0ad68d-c0a6-46e4-a7f8-2b4931dbed42 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:54.145070 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0ad68d-c0a6-46e4-a7f8-2b4931dbed42 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:54.151593 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca45ce7b-e433-45bb-a929-a1800d814f01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:54.152231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca45ce7b-e433-45bb-a929-a1800d814f01 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.152848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca45ce7b-e433-45bb-a929-a1800d814f01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.153537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca45ce7b-e433-45bb-a929-a1800d814f01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.154767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca45ce7b-e433-45bb-a929-a1800d814f01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:54.157287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca45ce7b-e433-45bb-a929-a1800d814f01 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:54.177024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 942/1875] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:54 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:54.189694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca45ce7b-e433-45bb-a929-a1800d814f01 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:54.222413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca45ce7b-e433-45bb-a929-a1800d814f01 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:54.245151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:54.245966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.246237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.246559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] PATH_INFO: `/v3/projects/1a07f2b75f7e478db20725ce61a1eab5/users/735d783343f5472a88885eadde38138f/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.251095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '737e742aabb34f32b1f6591a2841e107'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:54.262115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '737e742aabb34f32b1f6591a2841e107'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:54.262855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '737e742aabb34f32b1f6591a2841e107'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:54.263558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '737e742aabb34f32b1f6591a2841e107'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:54.264079 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a07f2b75f7e478db20725ce61a1eab5, user_id=735d783343f5472a88885eadde38138f, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:54.267561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:54.274671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9b084a26-529e-4010-8c14-0db176e63cad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '737e742aabb34f32b1f6591a2841e107'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:54.290414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 943/1876] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:54 2022] PUT /identity/v3/projects/1a07f2b75f7e478db20725ce61a1eab5/users/735d783343f5472a88885eadde38138f/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:54.294456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:54.297720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:54.299094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 934/1877] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:53 2022] DELETE /identity/v3/users/737e742aabb34f32b1f6591a2841e107 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:54.304731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7837aa-cfcc-43c9-8bff-c711c572311f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.304731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7837aa-cfcc-43c9-8bff-c711c572311f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.305057 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7837aa-cfcc-43c9-8bff-c711c572311f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.321712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:54.330524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ae7837aa-cfcc-43c9-8bff-c711c572311f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:54.331975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ae7837aa-cfcc-43c9-8bff-c711c572311f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:54.362982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:54.393821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaaf80d-757e-4df0-945e-ab2dba56d608 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:54.408923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 935/1878] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:54.420272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7837aa-cfcc-43c9-8bff-c711c572311f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:54.422704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ae7837aa-cfcc-43c9-8bff-c711c572311f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:54.467968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bed697-3c1d-473e-82ac-6da4cd7b8d7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:54.468881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bed697-3c1d-473e-82ac-6da4cd7b8d7a admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.469267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bed697-3c1d-473e-82ac-6da4cd7b8d7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.470006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bed697-3c1d-473e-82ac-6da4cd7b8d7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.471494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bed697-3c1d-473e-82ac-6da4cd7b8d7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:54.473817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bed697-3c1d-473e-82ac-6da4cd7b8d7a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:54.489477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 944/1879] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:54 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:54.495694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bed697-3c1d-473e-82ac-6da4cd7b8d7a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:54.507794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bed697-3c1d-473e-82ac-6da4cd7b8d7a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:54.511910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7837aa-cfcc-43c9-8bff-c711c572311f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:54.512789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1841c9-155f-408e-8212-f460b6faca85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:54.513566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1841c9-155f-408e-8212-f460b6faca85 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.513846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1841c9-155f-408e-8212-f460b6faca85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.514269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1841c9-155f-408e-8212-f460b6faca85 admin admin] PATH_INFO: `/v3/projects/1a07f2b75f7e478db20725ce61a1eab5/users/735d783343f5472a88885eadde38138f/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.516347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:54.516720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1841c9-155f-408e-8212-f460b6faca85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a07f2b75f7e478db20725ce61a1eab5, user_id=735d783343f5472a88885eadde38138f, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:54.517022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.517307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.517578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.519533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1841c9-155f-408e-8212-f460b6faca85 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:54.529704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:54.531721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:54.545373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 945/1880] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:54 2022] PUT /identity/v3/projects/1a07f2b75f7e478db20725ce61a1eab5/users/735d783343f5472a88885eadde38138f/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:54.549877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ef8d2e-97ff-44a8-a0f0-c70a9ffacd48 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.550334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ef8d2e-97ff-44a8-a0f0-c70a9ffacd48 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.550698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ef8d2e-97ff-44a8-a0f0-c70a9ffacd48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.552304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 936/1881] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:54.595989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e5ef8d2e-97ff-44a8-a0f0-c70a9ffacd48 None None] MFA Rules not processed for user `735d783343f5472a88885eadde38138f`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:54.598408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e5ef8d2e-97ff-44a8-a0f0-c70a9ffacd48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:54.711759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 946/1882] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:54 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:54.725139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:54.727949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:54.829963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6c1545-182f-4c68-b17b-62449519a697 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:54.926139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae12bd9-6925-47db-b05e-5e4ffb822260 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:54.926812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae12bd9-6925-47db-b05e-5e4ffb822260 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:54.927095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae12bd9-6925-47db-b05e-5e4ffb822260 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:54.927438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae12bd9-6925-47db-b05e-5e4ffb822260 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:54.932313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ef8d2e-97ff-44a8-a0f0-c70a9ffacd48 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:54.933976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e5ef8d2e-97ff-44a8-a0f0-c70a9ffacd48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:54.943040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae12bd9-6925-47db-b05e-5e4ffb822260 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:54.946740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae12bd9-6925-47db-b05e-5e4ffb822260 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:54.959774 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 937/1883] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:54 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:41:54.988390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:55.071277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ef8d2e-97ff-44a8-a0f0-c70a9ffacd48 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:55.079871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:55.080445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:55.088825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:55.088825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] PATH_INFO: `/v3/projects/2029d68d0ea74443964443dfb6e07aaa` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:55.090328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2029d68d0ea74443964443dfb6e07aaa)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:55.094431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:55.152813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2029d68d0ea74443964443dfb6e07aaa', 'request_id': 'req-e7a3980f-6e41-4142-afef-ea6b51f7cff1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.159234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2029d68d0ea74443964443dfb6e07aaa', 'request_id': 'req-e7a3980f-6e41-4142-afef-ea6b51f7cff1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.168233 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2029d68d0ea74443964443dfb6e07aaa', 'request_id': 'req-e7a3980f-6e41-4142-afef-ea6b51f7cff1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.190200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] The token cache is being invalidate because project 2029d68d0ea74443964443dfb6e07aaa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:55.190935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.191654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.193729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 947/1884] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:54 2022] DELETE /identity/v3/projects/2029d68d0ea74443964443dfb6e07aaa => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:55.197444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae12bd9-6925-47db-b05e-5e4ffb822260 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:55.214346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae12bd9-6925-47db-b05e-5e4ffb822260 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:55.259623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:55.306631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:55.307304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:55.308069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:55.308356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] PATH_INFO: `/v3/users/65cee7e355af4378ac1db4a1b946da8d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:55.309899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65cee7e355af4378ac1db4a1b946da8d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:55.316354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:55.387818 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65cee7e355af4378ac1db4a1b946da8d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.400091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65cee7e355af4378ac1db4a1b946da8d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.400772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65cee7e355af4378ac1db4a1b946da8d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.401582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65cee7e355af4378ac1db4a1b946da8d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.412237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65cee7e355af4378ac1db4a1b946da8d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.439143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 938/1885] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:55 2022] DELETE /identity/v3/users/65cee7e355af4378ac1db4a1b946da8d => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:55.735397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:55.736829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:55.755764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a3980f-6e41-4142-afef-ea6b51f7cff1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:55.820929 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:55.821576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:55.821861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:55.822138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] PATH_INFO: `/v3/projects/2a854d90b7214e6090921bb84aebf786` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:55.823345 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a854d90b7214e6090921bb84aebf786)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:55.825286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:55.874615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a854d90b7214e6090921bb84aebf786', 'request_id': 'req-943ca284-5f29-4a4b-9783-6def03e53e97'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.881225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a854d90b7214e6090921bb84aebf786', 'request_id': 'req-943ca284-5f29-4a4b-9783-6def03e53e97'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.883193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:55.891980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a854d90b7214e6090921bb84aebf786', 'request_id': 'req-943ca284-5f29-4a4b-9783-6def03e53e97'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.904978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] The token cache is being invalidate because project 2a854d90b7214e6090921bb84aebf786 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:55.905290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.905929 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:55.908259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 948/1886] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:55 2022] DELETE /identity/v3/projects/2a854d90b7214e6090921bb84aebf786 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:55.912112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:55.913530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:55.926467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f65584-698d-43d2-858f-4ac6dc3c9c14 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:55.984549 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:55.985193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:55.985474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:55.985750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] PATH_INFO: `/v3/users/4ab22342a008422b95ed73d2e310660a` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:55.986938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ab22342a008422b95ed73d2e310660a)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:55.988926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:56.038054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ab22342a008422b95ed73d2e310660a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.042715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:56.046277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ab22342a008422b95ed73d2e310660a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.047228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ab22342a008422b95ed73d2e310660a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.047838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ab22342a008422b95ed73d2e310660a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.056234 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e3b3d485-a3d0-45db-8dbe-bb3988941ee2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ab22342a008422b95ed73d2e310660a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.076547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 939/1887] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:55 2022] DELETE /identity/v3/users/4ab22342a008422b95ed73d2e310660a => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:56.381693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:56.383712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:56.399882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943ca284-5f29-4a4b-9783-6def03e53e97 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:56.466381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:56.467062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.467345 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.467754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] PATH_INFO: `/v3/projects/b12eebcd55084dcf8b3f50df5f124cac` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.468801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b12eebcd55084dcf8b3f50df5f124cac)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:56.471019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:56.524513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b12eebcd55084dcf8b3f50df5f124cac', 'request_id': 'req-e0f892e4-3655-4e9f-ba51-82efe94df6d5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.532806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b12eebcd55084dcf8b3f50df5f124cac', 'request_id': 'req-e0f892e4-3655-4e9f-ba51-82efe94df6d5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.543694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b12eebcd55084dcf8b3f50df5f124cac', 'request_id': 'req-e0f892e4-3655-4e9f-ba51-82efe94df6d5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.561549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] The token cache is being invalidate because project b12eebcd55084dcf8b3f50df5f124cac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:56.561849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.563732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e0f892e4-3655-4e9f-ba51-82efe94df6d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:56.568914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 949/1888] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:56 2022] DELETE /identity/v3/projects/b12eebcd55084dcf8b3f50df5f124cac => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:56.570578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e54b44-280e-4221-83c6-a7a7fc9e6f14 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.571748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e54b44-280e-4221-83c6-a7a7fc9e6f14 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.571748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e54b44-280e-4221-83c6-a7a7fc9e6f14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.592229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-20e54b44-280e-4221-83c6-a7a7fc9e6f14 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:56.593832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-20e54b44-280e-4221-83c6-a7a7fc9e6f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:56.659951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 940/1889] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:56.667813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feada395-e5d4-44f5-b97e-56ae26d049ed None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.668098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feada395-e5d4-44f5-b97e-56ae26d049ed None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.668399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feada395-e5d4-44f5-b97e-56ae26d049ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.694524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-feada395-e5d4-44f5-b97e-56ae26d049ed None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:56.696185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-feada395-e5d4-44f5-b97e-56ae26d049ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:56.704073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 950/1890] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:56.708414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e54b44-280e-4221-83c6-a7a7fc9e6f14 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:56.718117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e54b44-280e-4221-83c6-a7a7fc9e6f14 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:56.723123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:56.723714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.723990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.724256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.724891 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:56.725187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:56.726807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:56.738769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 941/1891] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:56.743046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feada395-e5d4-44f5-b97e-56ae26d049ed None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:56.753644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feada395-e5d4-44f5-b97e-56ae26d049ed None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:56.758319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091b075-8d9f-46aa-961e-ebed5bcbd634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:56.758957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091b075-8d9f-46aa-961e-ebed5bcbd634 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.759236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091b075-8d9f-46aa-961e-ebed5bcbd634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.759508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091b075-8d9f-46aa-961e-ebed5bcbd634 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.760149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c091b075-8d9f-46aa-961e-ebed5bcbd634 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:56.762042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c091b075-8d9f-46aa-961e-ebed5bcbd634 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:56.790400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 951/1892] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:56 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:56.794577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:56.805445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3805fb4c-a81c-4a7b-b532-db243569effe admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:56.864656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705e1573-3008-49cc-91dc-1ce03d1ee56b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:56.865267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705e1573-3008-49cc-91dc-1ce03d1ee56b admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.865550 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705e1573-3008-49cc-91dc-1ce03d1ee56b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.865826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705e1573-3008-49cc-91dc-1ce03d1ee56b admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.866534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705e1573-3008-49cc-91dc-1ce03d1ee56b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:56.868573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705e1573-3008-49cc-91dc-1ce03d1ee56b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:56.891145 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 942/1893] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:56 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:41:56.895254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091b075-8d9f-46aa-961e-ebed5bcbd634 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:56.905738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091b075-8d9f-46aa-961e-ebed5bcbd634 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:56.910927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0589914-82bc-4565-b0f5-c3889a8aef9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:56.911519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0589914-82bc-4565-b0f5-c3889a8aef9f admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.911796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0589914-82bc-4565-b0f5-c3889a8aef9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.912066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0589914-82bc-4565-b0f5-c3889a8aef9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.912701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0589914-82bc-4565-b0f5-c3889a8aef9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:56.914290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0589914-82bc-4565-b0f5-c3889a8aef9f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:56.924887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 952/1894] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:56.928918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705e1573-3008-49cc-91dc-1ce03d1ee56b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:56.939790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705e1573-3008-49cc-91dc-1ce03d1ee56b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:56.944477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64821515-f8a9-414c-b8a7-6658a65ddc8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:56.945079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64821515-f8a9-414c-b8a7-6658a65ddc8d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.945359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64821515-f8a9-414c-b8a7-6658a65ddc8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.945627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64821515-f8a9-414c-b8a7-6658a65ddc8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.946259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64821515-f8a9-414c-b8a7-6658a65ddc8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:56.947977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64821515-f8a9-414c-b8a7-6658a65ddc8d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:56.958843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 943/1895] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:56.963228 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0589914-82bc-4565-b0f5-c3889a8aef9f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:56.974141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0589914-82bc-4565-b0f5-c3889a8aef9f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:56.978820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:56.979498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:56.979774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:56.980047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] PATH_INFO: `/v3/projects/173a52b79dab4962a4aa380c26f97d2d/users/a08c9457231a447eb1c4130e68cfb31d/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:56.995154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] RBAC: Authorizing `identity:create_grant(project_id=173a52b79dab4962a4aa380c26f97d2d, user_id=a08c9457231a447eb1c4130e68cfb31d, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:56.997792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:57.012479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 953/1896] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:56 2022] PUT /identity/v3/projects/173a52b79dab4962a4aa380c26f97d2d/users/a08c9457231a447eb1c4130e68cfb31d/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:57.018471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b0314-d421-4f38-8ba2-55600d0e7003 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:57.018762 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b0314-d421-4f38-8ba2-55600d0e7003 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:57.019060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b0314-d421-4f38-8ba2-55600d0e7003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:57.038834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-0d6b0314-d421-4f38-8ba2-55600d0e7003 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:57.040044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0d6b0314-d421-4f38-8ba2-55600d0e7003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:57.072742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:57.106042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 944/1897] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:57.115747 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:57.117523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:57.133083 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:INSERT=4,keystone:DELETE=8,keystone:SELECT=88 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:57.169341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6b0314-d421-4f38-8ba2-55600d0e7003 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:57.174575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0d6b0314-d421-4f38-8ba2-55600d0e7003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:57.186371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9461177-fd35-4ba0-be98-534a3dd8a499 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:57.196738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b54847c-0257-474f-b52b-fb5aad98068c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:57.197379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b54847c-0257-474f-b52b-fb5aad98068c service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:57.197659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b54847c-0257-474f-b52b-fb5aad98068c service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:57.197944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b54847c-0257-474f-b52b-fb5aad98068c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:57.208295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6b0314-d421-4f38-8ba2-55600d0e7003 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:57.209184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b54847c-0257-474f-b52b-fb5aad98068c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:57.212060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b54847c-0257-474f-b52b-fb5aad98068c service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:57.223680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 954/1898] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:57.228845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:57.230033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:57.230568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:57.231048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] PATH_INFO: `/v3/users/2bb381599c3549e996ce8b27a4f42f69` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:57.235940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bb381599c3549e996ce8b27a4f42f69)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:57.238169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:57.302358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bb381599c3549e996ce8b27a4f42f69'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:57.310679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bb381599c3549e996ce8b27a4f42f69'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:57.311660 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bb381599c3549e996ce8b27a4f42f69'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:57.312415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bb381599c3549e996ce8b27a4f42f69'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:57.324352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bb381599c3549e996ce8b27a4f42f69'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:57.348410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 945/1899] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:57 2022] DELETE /identity/v3/users/2bb381599c3549e996ce8b27a4f42f69 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:57.353733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995898c-46e1-4edb-9f8c-887f08027e57 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:57.354020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995898c-46e1-4edb-9f8c-887f08027e57 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:57.354291 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d995898c-46e1-4edb-9f8c-887f08027e57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:57.377980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-d995898c-46e1-4edb-9f8c-887f08027e57 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:57.379912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d995898c-46e1-4edb-9f8c-887f08027e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:57.457203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 955/1900] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:57.468303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:57.469989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:57.534852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de745df-1ca5-4c0e-81b9-d304a99d2d78 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:57.539151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e2cad3-7113-45ba-96fb-fe88ebbcc8d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:57.539712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e2cad3-7113-45ba-96fb-fe88ebbcc8d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:57.539991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e2cad3-7113-45ba-96fb-fe88ebbcc8d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:57.540265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e2cad3-7113-45ba-96fb-fe88ebbcc8d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:57.552171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e2cad3-7113-45ba-96fb-fe88ebbcc8d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:57.555636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e2cad3-7113-45ba-96fb-fe88ebbcc8d8 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:57.569484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 946/1901] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:58.034368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d995898c-46e1-4edb-9f8c-887f08027e57 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:58.037782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d995898c-46e1-4edb-9f8c-887f08027e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:58.058060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d995898c-46e1-4edb-9f8c-887f08027e57 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:58.064096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:58.065052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:58.065336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:58.065610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] PATH_INFO: `/v3/projects/c0368a9c77754721b730abf25d485814` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:58.066820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0368a9c77754721b730abf25d485814)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:58.068755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:58.121739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95fe703-c917-47ef-b63c-cc1122395db9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:58.122268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95fe703-c917-47ef-b63c-cc1122395db9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:58.122347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95fe703-c917-47ef-b63c-cc1122395db9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:58.148412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0368a9c77754721b730abf25d485814', 'request_id': 'req-4fbe6d2d-213e-4a87-a6e9-3b563170176e'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.155432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0368a9c77754721b730abf25d485814', 'request_id': 'req-4fbe6d2d-213e-4a87-a6e9-3b563170176e'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.161809 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e95fe703-c917-47ef-b63c-cc1122395db9 None None] MFA Rules not processed for user `a08c9457231a447eb1c4130e68cfb31d`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:58.163541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e95fe703-c917-47ef-b63c-cc1122395db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:58.166538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0368a9c77754721b730abf25d485814', 'request_id': 'req-4fbe6d2d-213e-4a87-a6e9-3b563170176e'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.182710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] The token cache is being invalidate because project c0368a9c77754721b730abf25d485814 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:58.182923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.183602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.188410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 956/1902] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:58 2022] DELETE /identity/v3/projects/c0368a9c77754721b730abf25d485814 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:58.189482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:58.190929 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:58.221667 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:58.243256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:58.281216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 947/1903] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:41:58.287744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe6d2d-213e-4a87-a6e9-3b563170176e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:58.293282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:58.293993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:58.294225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:58.294544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] PATH_INFO: `/v3/users/735d783343f5472a88885eadde38138f` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:58.295837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=735d783343f5472a88885eadde38138f)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:58.296982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95fe703-c917-47ef-b63c-cc1122395db9 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:58.297870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:58.299181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e95fe703-c917-47ef-b63c-cc1122395db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:58.385953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '735d783343f5472a88885eadde38138f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.394388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95fe703-c917-47ef-b63c-cc1122395db9 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:58.396515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '735d783343f5472a88885eadde38138f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.397236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '735d783343f5472a88885eadde38138f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.397874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '735d783343f5472a88885eadde38138f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.398560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:58.399183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:58.399454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:58.399728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:58.402715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:58.407450 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d2c5d7e2-fd0c-4cf2-ac5d-a37236df3bb5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '735d783343f5472a88885eadde38138f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.416396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:58.418375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:58.430741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 948/1904] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:41:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:41:58.434801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 957/1905] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:41:58 2022] DELETE /identity/v3/users/735d783343f5472a88885eadde38138f => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:58.759782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:58.761958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:58.782181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1b5cf2-887d-4300-a1fc-2e599aa12d16 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:58.852765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:58.853414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:58.853695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:58.853973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] PATH_INFO: `/v3/projects/1a07f2b75f7e478db20725ce61a1eab5` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:58.855224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a07f2b75f7e478db20725ce61a1eab5)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:58.857163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:58.910719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a07f2b75f7e478db20725ce61a1eab5', 'request_id': 'req-2aecd384-641a-4a06-8262-0b8980616769'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.917433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a07f2b75f7e478db20725ce61a1eab5', 'request_id': 'req-2aecd384-641a-4a06-8262-0b8980616769'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.923669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a07f2b75f7e478db20725ce61a1eab5', 'request_id': 'req-2aecd384-641a-4a06-8262-0b8980616769'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.936797 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] The token cache is being invalidate because project 1a07f2b75f7e478db20725ce61a1eab5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:41:58.937059 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.937892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-2aecd384-641a-4a06-8262-0b8980616769 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:41:58.939531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 949/1906] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:41:58 2022] DELETE /identity/v3/projects/1a07f2b75f7e478db20725ce61a1eab5 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:58.944678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520e9e1-0644-486e-b601-437fa0c9af4a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:58.945171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520e9e1-0644-486e-b601-437fa0c9af4a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:58.945266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520e9e1-0644-486e-b601-437fa0c9af4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:58.966393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1520e9e1-0644-486e-b601-437fa0c9af4a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:58.967640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1520e9e1-0644-486e-b601-437fa0c9af4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:59.034017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 958/1907] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:59.041783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780226b0-a1c2-439e-b2c8-b2c3e135a521 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.042072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780226b0-a1c2-439e-b2c8-b2c3e135a521 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.042331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780226b0-a1c2-439e-b2c8-b2c3e135a521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.066089 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-780226b0-a1c2-439e-b2c8-b2c3e135a521 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:59.067415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-780226b0-a1c2-439e-b2c8-b2c3e135a521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:59.075636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 950/1908] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:59.080834 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1520e9e1-0644-486e-b601-437fa0c9af4a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.095083 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1520e9e1-0644-486e-b601-437fa0c9af4a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.100495 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.101116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.101406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.101687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.102342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:41:59.102675 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.104309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.116633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 959/1909] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:41:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:41:59.121011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780226b0-a1c2-439e-b2c8-b2c3e135a521 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.131586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780226b0-a1c2-439e-b2c8-b2c3e135a521 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.136421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7838b6-2a10-44b8-a7c7-7fea2b03e0f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.136994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7838b6-2a10-44b8-a7c7-7fea2b03e0f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.137299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7838b6-2a10-44b8-a7c7-7fea2b03e0f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.137572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7838b6-2a10-44b8-a7c7-7fea2b03e0f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.138335 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7838b6-2a10-44b8-a7c7-7fea2b03e0f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.140941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7838b6-2a10-44b8-a7c7-7fea2b03e0f3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.145650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:59.169836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 951/1910] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:59 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:59.174351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.184266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bda676-7c1c-4197-a322-82ceb7744c0f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.233576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:41:59.250684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b235a1-5264-4a80-a1ee-118977b964df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.251324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b235a1-5264-4a80-a1ee-118977b964df admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.251605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b235a1-5264-4a80-a1ee-118977b964df admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.251882 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b235a1-5264-4a80-a1ee-118977b964df admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.252557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b235a1-5264-4a80-a1ee-118977b964df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.254509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b235a1-5264-4a80-a1ee-118977b964df admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.277839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 960/1911] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:59 2022] POST /identity/v3/users => generated 287 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:59.282725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7838b6-2a10-44b8-a7c7-7fea2b03e0f3 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.293060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7838b6-2a10-44b8-a7c7-7fea2b03e0f3 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.297802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5061fd9f-b3d5-4b98-b5ad-c867a985a12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.298427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5061fd9f-b3d5-4b98-b5ad-c867a985a12d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.298714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5061fd9f-b3d5-4b98-b5ad-c867a985a12d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.298983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5061fd9f-b3d5-4b98-b5ad-c867a985a12d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.299623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5061fd9f-b3d5-4b98-b5ad-c867a985a12d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.301222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5061fd9f-b3d5-4b98-b5ad-c867a985a12d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.314730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 952/1912] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:41:59.319192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b235a1-5264-4a80-a1ee-118977b964df admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.330785 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b235a1-5264-4a80-a1ee-118977b964df admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.335505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20b81a8-075f-40d2-a9e5-124b9110e6c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.336111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b81a8-075f-40d2-a9e5-124b9110e6c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.336391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b81a8-075f-40d2-a9e5-124b9110e6c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.336664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20b81a8-075f-40d2-a9e5-124b9110e6c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.337306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20b81a8-075f-40d2-a9e5-124b9110e6c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.338963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20b81a8-075f-40d2-a9e5-124b9110e6c4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.354718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 961/1913] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:59.355899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5061fd9f-b3d5-4b98-b5ad-c867a985a12d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.366563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5061fd9f-b3d5-4b98-b5ad-c867a985a12d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.372051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6914fe3-a774-4dea-96d4-faf837947a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.372664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6914fe3-a774-4dea-96d4-faf837947a7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.372930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6914fe3-a774-4dea-96d4-faf837947a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.373225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6914fe3-a774-4dea-96d4-faf837947a7e admin admin] PATH_INFO: `/v3/projects/070556d5ac194eb58fcb89b143bcfc4f/users/05d03be22ed145b68160e145fd33f318/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.402426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6914fe3-a774-4dea-96d4-faf837947a7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=070556d5ac194eb58fcb89b143bcfc4f, user_id=05d03be22ed145b68160e145fd33f318, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.405144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6914fe3-a774-4dea-96d4-faf837947a7e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.418803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 953/1914] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:59 2022] PUT /identity/v3/projects/070556d5ac194eb58fcb89b143bcfc4f/users/05d03be22ed145b68160e145fd33f318/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:59.430192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96aa20e-8ebc-4361-8084-db15b78583f3 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.430512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96aa20e-8ebc-4361-8084-db15b78583f3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.430793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96aa20e-8ebc-4361-8084-db15b78583f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.459933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-a96aa20e-8ebc-4361-8084-db15b78583f3 None None] MFA Rules not processed for user `05d03be22ed145b68160e145fd33f318`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:41:59.461169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a96aa20e-8ebc-4361-8084-db15b78583f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:41:59.527801 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 962/1915] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:59 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:41:59.532942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6914fe3-a774-4dea-96d4-faf837947a7e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.597025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6914fe3-a774-4dea-96d4-faf837947a7e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.601689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772b0a03-8cdd-49d2-99c1-4b27fd51a3ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.602295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b0a03-8cdd-49d2-99c1-4b27fd51a3ee admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.602606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b0a03-8cdd-49d2-99c1-4b27fd51a3ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.602901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b0a03-8cdd-49d2-99c1-4b27fd51a3ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.603544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772b0a03-8cdd-49d2-99c1-4b27fd51a3ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.605475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772b0a03-8cdd-49d2-99c1-4b27fd51a3ee admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.634277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 954/1916] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:41:59 2022] POST /identity/v3/projects => generated 373 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:59.638428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96aa20e-8ebc-4361-8084-db15b78583f3 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.649720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96aa20e-8ebc-4361-8084-db15b78583f3 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.711215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd05236-4542-4d8f-8625-46a5e6a693f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.711826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd05236-4542-4d8f-8625-46a5e6a693f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.712107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd05236-4542-4d8f-8625-46a5e6a693f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.712383 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd05236-4542-4d8f-8625-46a5e6a693f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.713032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd05236-4542-4d8f-8625-46a5e6a693f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.714972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd05236-4542-4d8f-8625-46a5e6a693f8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.736843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 963/1917] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:41:59 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:41:59.741409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772b0a03-8cdd-49d2-99c1-4b27fd51a3ee admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.753320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772b0a03-8cdd-49d2-99c1-4b27fd51a3ee admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.758459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0bd441-6d95-40e5-8556-3132709ac8b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.759111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0bd441-6d95-40e5-8556-3132709ac8b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.759395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0bd441-6d95-40e5-8556-3132709ac8b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.759788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0bd441-6d95-40e5-8556-3132709ac8b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.760450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0bd441-6d95-40e5-8556-3132709ac8b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.762065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0bd441-6d95-40e5-8556-3132709ac8b6 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.774447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 955/1918] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:59.779125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd05236-4542-4d8f-8625-46a5e6a693f8 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.790958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd05236-4542-4d8f-8625-46a5e6a693f8 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.795621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbd5097-9e4e-444e-8930-b392fa7f83c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.796285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbd5097-9e4e-444e-8930-b392fa7f83c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.797142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbd5097-9e4e-444e-8930-b392fa7f83c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.797425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbd5097-9e4e-444e-8930-b392fa7f83c9 admin admin] PATH_INFO: `/v3/projects/6c0f9f75eace4dbe864dc16dafa042b2/users/b46c38a1eaae4d2e88d7422cf4874848/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.819438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbd5097-9e4e-444e-8930-b392fa7f83c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c0f9f75eace4dbe864dc16dafa042b2, user_id=b46c38a1eaae4d2e88d7422cf4874848, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.822612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbd5097-9e4e-444e-8930-b392fa7f83c9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.839576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 964/1919] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:59 2022] PUT /identity/v3/projects/6c0f9f75eace4dbe864dc16dafa042b2/users/b46c38a1eaae4d2e88d7422cf4874848/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:41:59.844443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0bd441-6d95-40e5-8556-3132709ac8b6 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.854875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0bd441-6d95-40e5-8556-3132709ac8b6 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.915843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93ff542-5d62-4857-a55f-895c83322ed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.916470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ff542-5d62-4857-a55f-895c83322ed2 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.916748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ff542-5d62-4857-a55f-895c83322ed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.917018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ff542-5d62-4857-a55f-895c83322ed2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.917665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93ff542-5d62-4857-a55f-895c83322ed2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.919340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93ff542-5d62-4857-a55f-895c83322ed2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.931027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 956/1920] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:41:59.935428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbd5097-9e4e-444e-8930-b392fa7f83c9 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:41:59.948315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbd5097-9e4e-444e-8930-b392fa7f83c9 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:41:59.953341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:41:59.954036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.954316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.954766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] PATH_INFO: `/v3/projects/6c0f9f75eace4dbe864dc16dafa042b2/users/b46c38a1eaae4d2e88d7422cf4874848/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:41:59.956773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c0f9f75eace4dbe864dc16dafa042b2, user_id=b46c38a1eaae4d2e88d7422cf4874848, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:41:59.959544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:41:59.971691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 965/1921] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:41:59 2022] PUT /identity/v3/projects/6c0f9f75eace4dbe864dc16dafa042b2/users/b46c38a1eaae4d2e88d7422cf4874848/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:41:59.983372 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc428d97-1b4e-40e8-a731-d91cb544ada9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:41:59.983663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc428d97-1b4e-40e8-a731-d91cb544ada9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:41:59.983930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc428d97-1b4e-40e8-a731-d91cb544ada9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:00.013570 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-cc428d97-1b4e-40e8-a731-d91cb544ada9 None None] MFA Rules not processed for user `b46c38a1eaae4d2e88d7422cf4874848`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:00.014817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-cc428d97-1b4e-40e8-a731-d91cb544ada9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:00.023599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:00.091224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 957/1922] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:41:59 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:00.106912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:00.108884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:00.120729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:00.177352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a4175c-0b9c-49f3-bb60-5a76a0dbea7e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:00.183614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36c526c-46e3-4668-80cf-5ef28161ebd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:00.184674 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c526c-46e3-4668-80cf-5ef28161ebd4 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:00.185174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c526c-46e3-4668-80cf-5ef28161ebd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:00.185654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36c526c-46e3-4668-80cf-5ef28161ebd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:00.203496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c526c-46e3-4668-80cf-5ef28161ebd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:00.206146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36c526c-46e3-4668-80cf-5ef28161ebd4 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:00.218116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 966/1923] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:00 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:00.296205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc428d97-1b4e-40e8-a731-d91cb544ada9 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:00.305845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc428d97-1b4e-40e8-a731-d91cb544ada9 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:00.309688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:00.310308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:00.310606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:00.310930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:00.370062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:00.371645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:00.381592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 958/1924] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:02.007956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1593cf5c-d4c6-4c61-9217-77a55cc8cdc2 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:02.008516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1593cf5c-d4c6-4c61-9217-77a55cc8cdc2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:02.008644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1593cf5c-d4c6-4c61-9217-77a55cc8cdc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:02.029272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1593cf5c-d4c6-4c61-9217-77a55cc8cdc2 None None] MFA Rules not processed for user `a08c9457231a447eb1c4130e68cfb31d`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:02.030685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1593cf5c-d4c6-4c61-9217-77a55cc8cdc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:02.092366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 967/1925] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:09.950620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:09.952052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:10.047337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b0895b-1b4b-47bc-a513-f869173b8516 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:10.053055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:10.053719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:10.054018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:10.054616 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] PATH_INFO: `/v3/users/830dcc8551dc4b3cba5af5569253a1fb` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:10.055475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] RBAC: Authorizing `identity:delete_user(user_id=830dcc8551dc4b3cba5af5569253a1fb)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:10.057886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:10.115522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:10.118902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '830dcc8551dc4b3cba5af5569253a1fb'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.128841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '830dcc8551dc4b3cba5af5569253a1fb'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.129587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '830dcc8551dc4b3cba5af5569253a1fb'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.130215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '830dcc8551dc4b3cba5af5569253a1fb'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.139061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '830dcc8551dc4b3cba5af5569253a1fb'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.158964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 959/1926] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:42:09 2022] DELETE /identity/v3/users/830dcc8551dc4b3cba5af5569253a1fb => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:42:10.163661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cf3cf-769e-4a28-b2c8-ffa883c047e7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:10.164169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cf3cf-769e-4a28-b2c8-ffa883c047e7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:10.164328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33cf3cf-769e-4a28-b2c8-ffa883c047e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:10.184288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e33cf3cf-769e-4a28-b2c8-ffa883c047e7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:10.186266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e33cf3cf-769e-4a28-b2c8-ffa883c047e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:10.267348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 968/1927] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:10.277342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:10.278771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:10.340054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b36449-9cce-46d8-af37-bccbf3c0c147 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:10.344673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447c3c23-b788-4b57-8cf3-994b5219ab27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:10.345545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c3c23-b788-4b57-8cf3-994b5219ab27 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:10.345846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c3c23-b788-4b57-8cf3-994b5219ab27 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:10.346095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c3c23-b788-4b57-8cf3-994b5219ab27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:10.356864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447c3c23-b788-4b57-8cf3-994b5219ab27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:10.358353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447c3c23-b788-4b57-8cf3-994b5219ab27 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:10.369155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 960/1928] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:10.748576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33cf3cf-769e-4a28-b2c8-ffa883c047e7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:10.750127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e33cf3cf-769e-4a28-b2c8-ffa883c047e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:10.765898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33cf3cf-769e-4a28-b2c8-ffa883c047e7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:10.770532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:10.771250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:10.771531 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:10.771803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] PATH_INFO: `/v3/projects/8653edd746b44c1eb38d97d3c138933d` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:10.772929 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8653edd746b44c1eb38d97d3c138933d)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:10.774887 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:10.829946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8653edd746b44c1eb38d97d3c138933d', 'request_id': 'req-b56e33d4-3a46-4246-8d03-f040111a200f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.831220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:10.838853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8653edd746b44c1eb38d97d3c138933d', 'request_id': 'req-b56e33d4-3a46-4246-8d03-f040111a200f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.850098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8653edd746b44c1eb38d97d3c138933d', 'request_id': 'req-b56e33d4-3a46-4246-8d03-f040111a200f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.867307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] The token cache is being invalidate because project 8653edd746b44c1eb38d97d3c138933d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:42:10.867602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.868270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b56e33d4-3a46-4246-8d03-f040111a200f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:10.869780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 969/1929] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:42:10 2022] DELETE /identity/v3/projects/8653edd746b44c1eb38d97d3c138933d => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:10.876134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d117e74d-b38e-470e-b49f-fecae15839ab None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:10.876422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d117e74d-b38e-470e-b49f-fecae15839ab None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:10.877033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d117e74d-b38e-470e-b49f-fecae15839ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:10.898507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d117e74d-b38e-470e-b49f-fecae15839ab None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:10.899874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d117e74d-b38e-470e-b49f-fecae15839ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:10.972422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 961/1930] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:42:10.980127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e739918-cfc9-4a07-8fec-9d745f815a41 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:10.980412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e739918-cfc9-4a07-8fec-9d745f815a41 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:10.980687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e739918-cfc9-4a07-8fec-9d745f815a41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:11.001119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5e739918-cfc9-4a07-8fec-9d745f815a41 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:11.002238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5e739918-cfc9-4a07-8fec-9d745f815a41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:11.010086 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 970/1931] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:11.014282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d117e74d-b38e-470e-b49f-fecae15839ab None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:11.025470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d117e74d-b38e-470e-b49f-fecae15839ab None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:11.030062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:11.031244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:11.031552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:11.031824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:11.032465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:42:11.032763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:11.034359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:11.046957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 962/1932] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:42:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:42:11.052513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e739918-cfc9-4a07-8fec-9d745f815a41 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:11.065235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e739918-cfc9-4a07-8fec-9d745f815a41 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:11.069908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0d1ca5-31be-4ff9-ad41-38335537396d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:11.070576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d1ca5-31be-4ff9-ad41-38335537396d admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:11.070854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d1ca5-31be-4ff9-ad41-38335537396d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:11.071128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d1ca5-31be-4ff9-ad41-38335537396d admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:11.071774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0d1ca5-31be-4ff9-ad41-38335537396d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:11.073692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0d1ca5-31be-4ff9-ad41-38335537396d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:11.109188 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 971/1933] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:42:11 2022] POST /identity/v3/projects => generated 367 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:11.110752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:11.121815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed929c-bd0e-45e3-829d-18ced5e73b8e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:11.176056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:11.187615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96893e0-96aa-445d-87ce-52742065ca67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:11.189648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96893e0-96aa-445d-87ce-52742065ca67 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:11.189943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96893e0-96aa-445d-87ce-52742065ca67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:11.190220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96893e0-96aa-445d-87ce-52742065ca67 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:11.190905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96893e0-96aa-445d-87ce-52742065ca67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:11.192854 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96893e0-96aa-445d-87ce-52742065ca67 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:11.215132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 963/1934] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:42:11 2022] POST /identity/v3/users => generated 284 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:11.219733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0d1ca5-31be-4ff9-ad41-38335537396d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:11.231257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0d1ca5-31be-4ff9-ad41-38335537396d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:11.236034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3130ac-2f1e-4bf7-8ef6-d4c259d7ac00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:11.236642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3130ac-2f1e-4bf7-8ef6-d4c259d7ac00 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:11.236996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3130ac-2f1e-4bf7-8ef6-d4c259d7ac00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:11.237270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3130ac-2f1e-4bf7-8ef6-d4c259d7ac00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:11.237961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3130ac-2f1e-4bf7-8ef6-d4c259d7ac00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:11.239603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3130ac-2f1e-4bf7-8ef6-d4c259d7ac00 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:11.253498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 972/1935] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:11 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:42:11.258265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96893e0-96aa-445d-87ce-52742065ca67 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:11.271890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96893e0-96aa-445d-87ce-52742065ca67 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:11.281854 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e88288-64dc-42c3-b494-a5f6bb027bf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:11.282536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e88288-64dc-42c3-b494-a5f6bb027bf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:11.282812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e88288-64dc-42c3-b494-a5f6bb027bf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:11.283084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e88288-64dc-42c3-b494-a5f6bb027bf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:11.283737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e88288-64dc-42c3-b494-a5f6bb027bf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:11.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e88288-64dc-42c3-b494-a5f6bb027bf1 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:11.299733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 964/1936] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:11 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:11.302976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3130ac-2f1e-4bf7-8ef6-d4c259d7ac00 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:11.315372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3130ac-2f1e-4bf7-8ef6-d4c259d7ac00 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:11.331614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:11.332297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:11.333026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:11.333370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] PATH_INFO: `/v3/projects/b403ad5109174ed0967e2c657f3298b9/users/9b938d369eda43bf9258c9103676e5b7/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:11.354457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b403ad5109174ed0967e2c657f3298b9, user_id=9b938d369eda43bf9258c9103676e5b7, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:11.355181 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:11.371626 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 973/1937] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:11 2022] PUT /identity/v3/projects/b403ad5109174ed0967e2c657f3298b9/users/9b938d369eda43bf9258c9103676e5b7/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:42:11.382315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4391a-efcc-43f3-9fd3-498878144681 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:11.382663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4391a-efcc-43f3-9fd3-498878144681 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:11.382964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4391a-efcc-43f3-9fd3-498878144681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:11.414993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-66a4391a-efcc-43f3-9fd3-498878144681 None None] MFA Rules not processed for user `9b938d369eda43bf9258c9103676e5b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:11.416233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-66a4391a-efcc-43f3-9fd3-498878144681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:11.488864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 965/1938] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:11 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:19.707930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:19.709655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:19.793843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70904aed-f109-4d47-a831-569f35596030 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:19.798046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df51226a-3c76-4806-9306-f450a2908164 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e3f6be584e34bfb9e430c7756d071d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:19.798725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df51226a-3c76-4806-9306-f450a2908164 service glance] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:19.799003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df51226a-3c76-4806-9306-f450a2908164 service glance] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:19.799289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df51226a-3c76-4806-9306-f450a2908164 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:19.808396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df51226a-3c76-4806-9306-f450a2908164 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:19.809757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df51226a-3c76-4806-9306-f450a2908164 service glance] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:19.819602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 974/1939] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:19 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:21.485975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=42,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:21.637007 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aed2da-ccdc-45a4-aca5-b56a5571e47a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:21.637290 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aed2da-ccdc-45a4-aca5-b56a5571e47a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:21.637561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aed2da-ccdc-45a4-aca5-b56a5571e47a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:21.657811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-f5aed2da-ccdc-45a4-aca5-b56a5571e47a None None] MFA Rules not processed for user `a08c9457231a447eb1c4130e68cfb31d`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:21.659152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f5aed2da-ccdc-45a4-aca5-b56a5571e47a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:21.735497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 966/1940] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:29.815238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:DELETE=3,keystone:SELECT=68,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:31.728031 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=25 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:44.854345 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb4f5a9-c80a-442d-beca-88534e44f08d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:44.855113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb4f5a9-c80a-442d-beca-88534e44f08d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:44.855325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb4f5a9-c80a-442d-beca-88534e44f08d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:44.878258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0eb4f5a9-c80a-442d-beca-88534e44f08d None None] MFA Rules not processed for user `6f8725c4d05b4dd79b05b43fba15980b`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:44.879596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0eb4f5a9-c80a-442d-beca-88534e44f08d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:44.944900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 975/1941] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:45.974086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aed2da-ccdc-45a4-aca5-b56a5571e47a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:45.977055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f5aed2da-ccdc-45a4-aca5-b56a5571e47a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:46.104595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aed2da-ccdc-45a4-aca5-b56a5571e47a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:46.111752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:46.112410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:46.112739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:46.113016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] PATH_INFO: `/v3/users/05d03be22ed145b68160e145fd33f318` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:46.114166 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] RBAC: Authorizing `identity:delete_user(user_id=05d03be22ed145b68160e145fd33f318)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:46.116370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:46.173500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05d03be22ed145b68160e145fd33f318'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.181442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05d03be22ed145b68160e145fd33f318'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.182184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05d03be22ed145b68160e145fd33f318'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.182820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05d03be22ed145b68160e145fd33f318'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.192326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05d03be22ed145b68160e145fd33f318'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.214705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 967/1942] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:42:45 2022] DELETE /identity/v3/users/05d03be22ed145b68160e145fd33f318 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:46.220816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a1864-a9ec-4fb0-b429-d0b0480eb65c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:46.221319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a1864-a9ec-4fb0-b429-d0b0480eb65c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:46.221553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a1864-a9ec-4fb0-b429-d0b0480eb65c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:46.243151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-cc7a1864-a9ec-4fb0-b429-d0b0480eb65c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:46.244481 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cc7a1864-a9ec-4fb0-b429-d0b0480eb65c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:46.309420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 976/1943] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:42:46.321757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:46.324095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:46.384954 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7a1864-a9ec-4fb0-b429-d0b0480eb65c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:46.386590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cc7a1864-a9ec-4fb0-b429-d0b0480eb65c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:46.386925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60487969-be64-4896-9f6d-032c48a8565f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:46.391147 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2067f571-f47d-47a3-8742-70452f2cadf7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:46.391798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2067f571-f47d-47a3-8742-70452f2cadf7 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:46.392084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2067f571-f47d-47a3-8742-70452f2cadf7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:46.392353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2067f571-f47d-47a3-8742-70452f2cadf7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:46.402410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7a1864-a9ec-4fb0-b429-d0b0480eb65c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:46.403066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2067f571-f47d-47a3-8742-70452f2cadf7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:46.406549 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2067f571-f47d-47a3-8742-70452f2cadf7 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:46.408707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:46.409411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:46.409693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:46.410187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] PATH_INFO: `/v3/users/d185d73daa1a440088b4c6fa7134dce0` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:46.411446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d185d73daa1a440088b4c6fa7134dce0)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:46.414018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:46.418303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 968/1944] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:46.489109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd185d73daa1a440088b4c6fa7134dce0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.518040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd185d73daa1a440088b4c6fa7134dce0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.519158 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd185d73daa1a440088b4c6fa7134dce0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.520350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd185d73daa1a440088b4c6fa7134dce0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.531780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd185d73daa1a440088b4c6fa7134dce0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.562097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 977/1945] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:42:46 2022] DELETE /identity/v3/users/d185d73daa1a440088b4c6fa7134dce0 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:42:46.564100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6855fc-b825-4c3b-968b-8c15188b6a67 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:46.565187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6855fc-b825-4c3b-968b-8c15188b6a67 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:46.566541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6855fc-b825-4c3b-968b-8c15188b6a67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:46.601877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-9b6855fc-b825-4c3b-968b-8c15188b6a67 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:46.603748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9b6855fc-b825-4c3b-968b-8c15188b6a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:46.626069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:46.694665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 969/1946] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:46.705765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:46.707583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:46.734301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:46.790743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646e2d43-cde6-474e-beac-4d972a6b488a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:46.795328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709c5a99-1db9-4e73-bd1f-16af49fde1ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:46.796065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709c5a99-1db9-4e73-bd1f-16af49fde1ae service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:46.796426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709c5a99-1db9-4e73-bd1f-16af49fde1ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:46.796705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709c5a99-1db9-4e73-bd1f-16af49fde1ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:46.799708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6855fc-b825-4c3b-968b-8c15188b6a67 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:46.801321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9b6855fc-b825-4c3b-968b-8c15188b6a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:46.810185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709c5a99-1db9-4e73-bd1f-16af49fde1ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:46.812795 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709c5a99-1db9-4e73-bd1f-16af49fde1ae service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:46.824822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 978/1947] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:46.830513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6855fc-b825-4c3b-968b-8c15188b6a67 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:46.837149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:46.838028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:46.838511 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:46.838894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] PATH_INFO: `/v3/projects/070556d5ac194eb58fcb89b143bcfc4f` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:46.840441 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] RBAC: Authorizing `identity:delete_project(project_id=070556d5ac194eb58fcb89b143bcfc4f)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:46.843628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:46.906622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '070556d5ac194eb58fcb89b143bcfc4f', 'request_id': 'req-d7940945-3f2d-4124-a9aa-1f5e474eee7b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.914996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '070556d5ac194eb58fcb89b143bcfc4f', 'request_id': 'req-d7940945-3f2d-4124-a9aa-1f5e474eee7b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.923094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '070556d5ac194eb58fcb89b143bcfc4f', 'request_id': 'req-d7940945-3f2d-4124-a9aa-1f5e474eee7b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.937205 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] The token cache is being invalidate because project 070556d5ac194eb58fcb89b143bcfc4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:42:46.937492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.938174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:46.939710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 970/1948] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:42:46 2022] DELETE /identity/v3/projects/070556d5ac194eb58fcb89b143bcfc4f => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:46.943628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709c5a99-1db9-4e73-bd1f-16af49fde1ae service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:46.955489 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709c5a99-1db9-4e73-bd1f-16af49fde1ae service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:47.024908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:47.025567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:47.025848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:47.026143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] PATH_INFO: `/v3/users/b46c38a1eaae4d2e88d7422cf4874848` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:47.027604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b46c38a1eaae4d2e88d7422cf4874848)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:47.029397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:47.187000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b46c38a1eaae4d2e88d7422cf4874848'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.204549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b46c38a1eaae4d2e88d7422cf4874848'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.207452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b46c38a1eaae4d2e88d7422cf4874848'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.208071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b46c38a1eaae4d2e88d7422cf4874848'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.247607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9afca6a3-3837-4d81-8b76-a4ea5e58aab3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b46c38a1eaae4d2e88d7422cf4874848'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.303504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 979/1949] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:42:46 2022] DELETE /identity/v3/users/b46c38a1eaae4d2e88d7422cf4874848 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:42:47.506413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:47.508060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:47.538105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7940945-3f2d-4124-a9aa-1f5e474eee7b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:47.682569 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:47.683403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:47.683682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:47.684295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] PATH_INFO: `/v3/projects/370471a6d2ef476db1f8dc94af08a88d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:47.685892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] RBAC: Authorizing `identity:delete_project(project_id=370471a6d2ef476db1f8dc94af08a88d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:47.690960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:47.763142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:47.765251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '370471a6d2ef476db1f8dc94af08a88d', 'request_id': 'req-4caa94cc-2824-4803-84b0-a3242ff4495b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.776612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '370471a6d2ef476db1f8dc94af08a88d', 'request_id': 'req-4caa94cc-2824-4803-84b0-a3242ff4495b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.784557 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '370471a6d2ef476db1f8dc94af08a88d', 'request_id': 'req-4caa94cc-2824-4803-84b0-a3242ff4495b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.803832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] The token cache is being invalidate because project 370471a6d2ef476db1f8dc94af08a88d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:42:47.804130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.804784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4caa94cc-2824-4803-84b0-a3242ff4495b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:47.807484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 971/1950] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:42:47 2022] DELETE /identity/v3/projects/370471a6d2ef476db1f8dc94af08a88d => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:47.812926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c5f59-e076-4d04-9bbd-26bf496c880d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:47.813216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c5f59-e076-4d04-9bbd-26bf496c880d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:47.813493 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c5f59-e076-4d04-9bbd-26bf496c880d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:47.837094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8c1c5f59-e076-4d04-9bbd-26bf496c880d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:47.838727 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8c1c5f59-e076-4d04-9bbd-26bf496c880d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:47.882125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:47.920787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 980/1951] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:47.929034 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54266007-f4bc-44a0-b8e6-5b8ba4ab5f42 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:47.929319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54266007-f4bc-44a0-b8e6-5b8ba4ab5f42 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:47.929597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54266007-f4bc-44a0-b8e6-5b8ba4ab5f42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:47.953672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-54266007-f4bc-44a0-b8e6-5b8ba4ab5f42 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:47.954948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-54266007-f4bc-44a0-b8e6-5b8ba4ab5f42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:47.963456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 972/1952] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:42:47.968175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c5f59-e076-4d04-9bbd-26bf496c880d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:47.978425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c5f59-e076-4d04-9bbd-26bf496c880d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:47.983663 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:47.984280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:47.984559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:47.984831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:47.985485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:42:47.985846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:47.987551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.001545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 981/1953] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:42:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:42:48.006545 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54266007-f4bc-44a0-b8e6-5b8ba4ab5f42 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.017524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54266007-f4bc-44a0-b8e6-5b8ba4ab5f42 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.017932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.020864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:48.023346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf01ff9-cb8d-450a-9017-4aac91f54e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.024139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf01ff9-cb8d-450a-9017-4aac91f54e8f admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.024563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf01ff9-cb8d-450a-9017-4aac91f54e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.024989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf01ff9-cb8d-450a-9017-4aac91f54e8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.025859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf01ff9-cb8d-450a-9017-4aac91f54e8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.028416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf01ff9-cb8d-450a-9017-4aac91f54e8f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.036219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c89544-25d9-4772-ad31-904309f9684c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.041923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.042736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.043109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.043439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] PATH_INFO: `/v3/projects/6c0f9f75eace4dbe864dc16dafa042b2` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.045195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c0f9f75eace4dbe864dc16dafa042b2)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.047787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.068171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 973/1954] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:42:48 2022] POST /identity/v3/projects => generated 383 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:48.072884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf01ff9-cb8d-450a-9017-4aac91f54e8f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.110505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c0f9f75eace4dbe864dc16dafa042b2', 'request_id': 'req-4f427714-efe8-41f7-bc24-b36642524ec4'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:48.118346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c0f9f75eace4dbe864dc16dafa042b2', 'request_id': 'req-4f427714-efe8-41f7-bc24-b36642524ec4'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:48.132535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c0f9f75eace4dbe864dc16dafa042b2', 'request_id': 'req-4f427714-efe8-41f7-bc24-b36642524ec4'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:48.147798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] The token cache is being invalidate because project 6c0f9f75eace4dbe864dc16dafa042b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:42:48.148119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:48.148835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4f427714-efe8-41f7-bc24-b36642524ec4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:48.150534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 982/1955] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:42:48 2022] DELETE /identity/v3/projects/6c0f9f75eace4dbe864dc16dafa042b2 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:42:48.156115 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f12be-301e-4194-8026-f29de709db52 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.156571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f12be-301e-4194-8026-f29de709db52 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.156878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f12be-301e-4194-8026-f29de709db52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.164951 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf01ff9-cb8d-450a-9017-4aac91f54e8f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.184490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c16f12be-301e-4194-8026-f29de709db52 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:48.186438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c16f12be-301e-4194-8026-f29de709db52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:48.245905 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.246356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.246703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.246976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.247826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.249900 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.256639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 983/1956] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:48.266991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e439580e-36f0-44f1-b5aa-93cedd248dcc None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.267355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e439580e-36f0-44f1-b5aa-93cedd248dcc None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.267673 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e439580e-36f0-44f1-b5aa-93cedd248dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.276155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 974/1957] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:42:48 2022] POST /identity/v3/users => generated 292 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:48.280882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.283253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:48.295875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcad139-0791-47d3-92dd-33e5854922fb admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.296252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e439580e-36f0-44f1-b5aa-93cedd248dcc None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:48.297536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e439580e-36f0-44f1-b5aa-93cedd248dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:48.301580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44d7a44-69fa-4d76-9c99-4ad00888bbcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.302221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44d7a44-69fa-4d76-9c99-4ad00888bbcc admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.302516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44d7a44-69fa-4d76-9c99-4ad00888bbcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.302799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44d7a44-69fa-4d76-9c99-4ad00888bbcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.303507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44d7a44-69fa-4d76-9c99-4ad00888bbcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.305223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44d7a44-69fa-4d76-9c99-4ad00888bbcc admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.306751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 984/1958] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:48.311260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e439580e-36f0-44f1-b5aa-93cedd248dcc None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.321885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 975/1959] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:48 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:42:48.323278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e439580e-36f0-44f1-b5aa-93cedd248dcc None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.326034 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44d7a44-69fa-4d76-9c99-4ad00888bbcc admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.331529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.332167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.332446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.332752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.333523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:42:48.333829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.335524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.339458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44d7a44-69fa-4d76-9c99-4ad00888bbcc admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.346722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14a8d40-add3-4de7-8859-3e12aaacf794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.348255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a8d40-add3-4de7-8859-3e12aaacf794 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.348823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a8d40-add3-4de7-8859-3e12aaacf794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.349467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a8d40-add3-4de7-8859-3e12aaacf794 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.350323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14a8d40-add3-4de7-8859-3e12aaacf794 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.350469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 985/1960] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:42:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:42:48.353004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14a8d40-add3-4de7-8859-3e12aaacf794 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.354655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.368711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58822974-2ed9-4a59-b7ea-51ded650a7db admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.374802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 976/1961] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:48 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:42:48.377554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14a8d40-add3-4de7-8859-3e12aaacf794 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.380138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80971ef5-6d10-4427-a65e-8242297b634a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.381104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80971ef5-6d10-4427-a65e-8242297b634a admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.381234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80971ef5-6d10-4427-a65e-8242297b634a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.381595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80971ef5-6d10-4427-a65e-8242297b634a admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.382473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80971ef5-6d10-4427-a65e-8242297b634a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.384823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80971ef5-6d10-4427-a65e-8242297b634a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.389041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14a8d40-add3-4de7-8859-3e12aaacf794 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.399173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af9afdc-1353-4cb2-90c1-7a27d91f0c01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.401355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af9afdc-1353-4cb2-90c1-7a27d91f0c01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.401355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af9afdc-1353-4cb2-90c1-7a27d91f0c01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.401621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af9afdc-1353-4cb2-90c1-7a27d91f0c01 admin admin] PATH_INFO: `/v3/projects/1235d8b054b3405595fca761f08b73ae/users/39de6a6fff7c4cc08cc62aa86182d71d/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.421083 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af9afdc-1353-4cb2-90c1-7a27d91f0c01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1235d8b054b3405595fca761f08b73ae, user_id=39de6a6fff7c4cc08cc62aa86182d71d, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.424120 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af9afdc-1353-4cb2-90c1-7a27d91f0c01 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.428499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 986/1962] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:42:48 2022] POST /identity/v3/projects => generated 371 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:42:48.429696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80971ef5-6d10-4427-a65e-8242297b634a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.436890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=91,keystone:DELETE=3,keystone:INSERT=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:48.448530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 977/1963] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:48 2022] PUT /identity/v3/projects/1235d8b054b3405595fca761f08b73ae/users/39de6a6fff7c4cc08cc62aa86182d71d/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:48.450476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:48.455997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324bb18d-82ce-4d2f-b659-6b7468caa715 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.456535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324bb18d-82ce-4d2f-b659-6b7468caa715 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.456907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324bb18d-82ce-4d2f-b659-6b7468caa715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.507673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-324bb18d-82ce-4d2f-b659-6b7468caa715 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:48.511273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-324bb18d-82ce-4d2f-b659-6b7468caa715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:48.585210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80971ef5-6d10-4427-a65e-8242297b634a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.693157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd5e21-e0f9-409e-96a3-fc54af43a740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.693788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd5e21-e0f9-409e-96a3-fc54af43a740 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.694121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd5e21-e0f9-409e-96a3-fc54af43a740 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.694401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bd5e21-e0f9-409e-96a3-fc54af43a740 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.695576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bd5e21-e0f9-409e-96a3-fc54af43a740 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.697969 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bd5e21-e0f9-409e-96a3-fc54af43a740 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.725177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 987/1964] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:42:48 2022] POST /identity/v3/users => generated 286 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:48.729538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd5e21-e0f9-409e-96a3-fc54af43a740 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.756101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 978/1965] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:48.756277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bd5e21-e0f9-409e-96a3-fc54af43a740 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.764792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324bb18d-82ce-4d2f-b659-6b7468caa715 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.766379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35549c04-ee5e-446c-987d-a730f14022c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.767083 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-324bb18d-82ce-4d2f-b659-6b7468caa715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:48.767204 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35549c04-ee5e-446c-987d-a730f14022c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.767377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35549c04-ee5e-446c-987d-a730f14022c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.768094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35549c04-ee5e-446c-987d-a730f14022c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.768911 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35549c04-ee5e-446c-987d-a730f14022c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.770815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35549c04-ee5e-446c-987d-a730f14022c6 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.786023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 988/1966] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:48 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:48.790533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35549c04-ee5e-446c-987d-a730f14022c6 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.808374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35549c04-ee5e-446c-987d-a730f14022c6 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.820430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f073f22-2e4f-472b-a744-95991b938484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.821116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f073f22-2e4f-472b-a744-95991b938484 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.821983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f073f22-2e4f-472b-a744-95991b938484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.821983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f073f22-2e4f-472b-a744-95991b938484 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.822292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f073f22-2e4f-472b-a744-95991b938484 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.827007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f073f22-2e4f-472b-a744-95991b938484 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.848313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 989/1967] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:48 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:48.853116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f073f22-2e4f-472b-a744-95991b938484 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:48.875024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f073f22-2e4f-472b-a744-95991b938484 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.888332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca612101-f7d6-4512-a404-ad326234531c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.889708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca612101-f7d6-4512-a404-ad326234531c admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.890089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca612101-f7d6-4512-a404-ad326234531c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.892560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca612101-f7d6-4512-a404-ad326234531c admin admin] PATH_INFO: `/v3/projects/b6c4100c21c44826a0c550533303c267/users/fe30915eb58e452aa8c86242d78dc281/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.907034 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324bb18d-82ce-4d2f-b659-6b7468caa715 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:48.916210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e26d9e-3e9c-4359-9601-188808143b83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:48.916857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e26d9e-3e9c-4359-9601-188808143b83 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.917130 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e26d9e-3e9c-4359-9601-188808143b83 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.918477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e26d9e-3e9c-4359-9601-188808143b83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:48.927165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca612101-f7d6-4512-a404-ad326234531c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6c4100c21c44826a0c550533303c267, user_id=fe30915eb58e452aa8c86242d78dc281, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.931169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca612101-f7d6-4512-a404-ad326234531c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.933516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e26d9e-3e9c-4359-9601-188808143b83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:48.935525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e26d9e-3e9c-4359-9601-188808143b83 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:48.949736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 979/1968] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:42:48.958314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 990/1969] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:48 2022] PUT /identity/v3/projects/b6c4100c21c44826a0c550533303c267/users/fe30915eb58e452aa8c86242d78dc281/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:48.977955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89cdf51-7bd6-40a1-9f8b-174c4f096793 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:48.978244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89cdf51-7bd6-40a1-9f8b-174c4f096793 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:48.979973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89cdf51-7bd6-40a1-9f8b-174c4f096793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:49.050551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c89cdf51-7bd6-40a1-9f8b-174c4f096793 None None] MFA Rules not processed for user `fe30915eb58e452aa8c86242d78dc281`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:49.051922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c89cdf51-7bd6-40a1-9f8b-174c4f096793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:49.170424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 980/1970] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:48 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:49.174668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca612101-f7d6-4512-a404-ad326234531c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:49.337332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca612101-f7d6-4512-a404-ad326234531c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:49.355023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8494da24-c844-469b-9cfb-d12749ca2455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:49.355916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494da24-c844-469b-9cfb-d12749ca2455 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:49.357149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494da24-c844-469b-9cfb-d12749ca2455 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:49.357599 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494da24-c844-469b-9cfb-d12749ca2455 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:49.359174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8494da24-c844-469b-9cfb-d12749ca2455 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:49.362553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8494da24-c844-469b-9cfb-d12749ca2455 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:49.368551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:49.431820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 991/1971] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:42:49 2022] POST /identity/v3/projects => generated 369 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:49.437829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89cdf51-7bd6-40a1-9f8b-174c4f096793 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:49.454215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89cdf51-7bd6-40a1-9f8b-174c4f096793 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:49.533973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:49.601087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd423187-0259-4338-b3cd-0fedb3a3048c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:49.601620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd423187-0259-4338-b3cd-0fedb3a3048c admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:49.602291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd423187-0259-4338-b3cd-0fedb3a3048c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:49.602291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd423187-0259-4338-b3cd-0fedb3a3048c admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:49.603516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd423187-0259-4338-b3cd-0fedb3a3048c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:49.605682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd423187-0259-4338-b3cd-0fedb3a3048c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:49.642521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 981/1972] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:42:49 2022] POST /identity/v3/users => generated 291 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:49.645535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8494da24-c844-469b-9cfb-d12749ca2455 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:49.671353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8494da24-c844-469b-9cfb-d12749ca2455 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:49.681114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ed9e1b-e162-413e-b103-d5d6b2a6210c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:49.682886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ed9e1b-e162-413e-b103-d5d6b2a6210c admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:49.683822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ed9e1b-e162-413e-b103-d5d6b2a6210c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:49.684597 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ed9e1b-e162-413e-b103-d5d6b2a6210c admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:49.686069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ed9e1b-e162-413e-b103-d5d6b2a6210c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:49.689178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ed9e1b-e162-413e-b103-d5d6b2a6210c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:49.721513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 992/1973] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:49 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:42:49.723885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd423187-0259-4338-b3cd-0fedb3a3048c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:49.746372 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd423187-0259-4338-b3cd-0fedb3a3048c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:49.757976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd2f513-5e51-433f-aefa-249d3865b067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:49.759602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd2f513-5e51-433f-aefa-249d3865b067 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:49.760105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd2f513-5e51-433f-aefa-249d3865b067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:49.760615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd2f513-5e51-433f-aefa-249d3865b067 admin admin] PATH_INFO: `/v3/projects/35d19bb82dbe4d84a8dc0a534a0328f1/users/089441a372bd4b7a8ed7aa55cdb30df6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:49.818917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd2f513-5e51-433f-aefa-249d3865b067 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35d19bb82dbe4d84a8dc0a534a0328f1, user_id=089441a372bd4b7a8ed7aa55cdb30df6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:49.822215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd2f513-5e51-433f-aefa-249d3865b067 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:49.849431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 982/1974] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:49 2022] PUT /identity/v3/projects/35d19bb82dbe4d84a8dc0a534a0328f1/users/089441a372bd4b7a8ed7aa55cdb30df6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:49.853424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ed9e1b-e162-413e-b103-d5d6b2a6210c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:49.865617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ed9e1b-e162-413e-b103-d5d6b2a6210c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:49.963515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fdcad1-d09a-491f-93ad-d647d862efa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:49.973603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fdcad1-d09a-491f-93ad-d647d862efa3 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:49.975044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fdcad1-d09a-491f-93ad-d647d862efa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:49.977191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fdcad1-d09a-491f-93ad-d647d862efa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:49.982646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fdcad1-d09a-491f-93ad-d647d862efa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:49.987314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fdcad1-d09a-491f-93ad-d647d862efa3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:50.019056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 993/1975] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:49 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:50.019878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd2f513-5e51-433f-aefa-249d3865b067 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:50.034926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd2f513-5e51-433f-aefa-249d3865b067 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:50.052343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:50.053180 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:50.053560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:50.053935 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] PATH_INFO: `/v3/projects/35d19bb82dbe4d84a8dc0a534a0328f1/users/089441a372bd4b7a8ed7aa55cdb30df6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:50.056218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35d19bb82dbe4d84a8dc0a534a0328f1, user_id=089441a372bd4b7a8ed7aa55cdb30df6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:50.061295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:50.086741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 983/1976] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:50 2022] PUT /identity/v3/projects/35d19bb82dbe4d84a8dc0a534a0328f1/users/089441a372bd4b7a8ed7aa55cdb30df6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:50.098266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c82f37-b0c8-48de-98eb-b49ab7fd0590 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:50.098831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c82f37-b0c8-48de-98eb-b49ab7fd0590 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:50.099298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c82f37-b0c8-48de-98eb-b49ab7fd0590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:50.166738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-57c82f37-b0c8-48de-98eb-b49ab7fd0590 None None] MFA Rules not processed for user `089441a372bd4b7a8ed7aa55cdb30df6`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:50.168022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-57c82f37-b0c8-48de-98eb-b49ab7fd0590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:50.319017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 994/1977] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:50 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:50.332732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:50.335613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:50.422070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783e57f2-782b-426c-af05-bf7eeb6bf682 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:50.426359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:50.426563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:50.426845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:50.427121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:50.438303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:50.440321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:50.452988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 984/1978] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:50 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:50.768392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6d38a6-4f1d-45b4-a53d-9ccd250794ec None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:50.769003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6d38a6-4f1d-45b4-a53d-9ccd250794ec None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:50.769296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6d38a6-4f1d-45b4-a53d-9ccd250794ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:50.820451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-be6d38a6-4f1d-45b4-a53d-9ccd250794ec None None] MFA Rules not processed for user `39de6a6fff7c4cc08cc62aa86182d71d`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:50.823336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-be6d38a6-4f1d-45b4-a53d-9ccd250794ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:50.926854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:51.009184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 995/1979] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:51.038880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:51.041675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:51.061690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7783ec2f-3f45-47d8-9738-be1563f4566c service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:51.068960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5820ea07-fd01-4cd5-8673-be562fcc44f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:51.069591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820ea07-fd01-4cd5-8673-be562fcc44f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:51.069872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820ea07-fd01-4cd5-8673-be562fcc44f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:51.070648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820ea07-fd01-4cd5-8673-be562fcc44f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:51.072994 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6d38a6-4f1d-45b4-a53d-9ccd250794ec None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:51.104120 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5820ea07-fd01-4cd5-8673-be562fcc44f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:51.106056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5820ea07-fd01-4cd5-8673-be562fcc44f6 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:51.130995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 985/1980] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:42:51.180339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6d38a6-4f1d-45b4-a53d-9ccd250794ec None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:51.187401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:51.188060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:51.188340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:51.188615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] PATH_INFO: `/v3/users/fe30915eb58e452aa8c86242d78dc281` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:51.190425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe30915eb58e452aa8c86242d78dc281)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:51.192595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:51.281605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe30915eb58e452aa8c86242d78dc281'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:51.295307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe30915eb58e452aa8c86242d78dc281'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:51.295307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe30915eb58e452aa8c86242d78dc281'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:51.295307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe30915eb58e452aa8c86242d78dc281'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:51.305990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe30915eb58e452aa8c86242d78dc281'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:51.334521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 996/1981] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:42:51 2022] DELETE /identity/v3/users/fe30915eb58e452aa8c86242d78dc281 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:51.339239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f1038-5d76-42fe-8e82-3eda6cf4c0dd None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:51.339618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f1038-5d76-42fe-8e82-3eda6cf4c0dd None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:51.339862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f1038-5d76-42fe-8e82-3eda6cf4c0dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:51.364736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c11f1038-5d76-42fe-8e82-3eda6cf4c0dd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:51.366035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c11f1038-5d76-42fe-8e82-3eda6cf4c0dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:51.468075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 986/1982] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:42:51.483813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:51.485416 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:51.562734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b62116-d2a7-4f17-afca-62bdb32de75f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:51.563052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b62116-d2a7-4f17-afca-62bdb32de75f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:51.563276 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b62116-d2a7-4f17-afca-62bdb32de75f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:51.599428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a5b62116-d2a7-4f17-afca-62bdb32de75f None None] MFA Rules not processed for user `d3f0d615187d4779bdd7da8c9f24c796`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:51.601249 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a5b62116-d2a7-4f17-afca-62bdb32de75f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:51.612483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:51.613491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8ea8ef-09c9-4fd4-9d1b-d70c3441f570 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:51.617495 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:51.618131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:51.618414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:51.618742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:51.629909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:51.631394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:51.643370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 997/1983] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:51.686516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 987/1984] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:51 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:42:52.271004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:52.273077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:52.288178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1843374-bb68-4063-9db7-e6e21216b729 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:52.295921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:52.297318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:52.297870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:52.298442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] PATH_INFO: `/v3/projects/b6c4100c21c44826a0c550533303c267` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:52.300633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6c4100c21c44826a0c550533303c267)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:52.304730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:52.307630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:52.387703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6c4100c21c44826a0c550533303c267', 'request_id': 'req-ea12840e-595c-4c27-94ab-631a157680d1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.396861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6c4100c21c44826a0c550533303c267', 'request_id': 'req-ea12840e-595c-4c27-94ab-631a157680d1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.406668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6c4100c21c44826a0c550533303c267', 'request_id': 'req-ea12840e-595c-4c27-94ab-631a157680d1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.420552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] The token cache is being invalidate because project b6c4100c21c44826a0c550533303c267 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:42:52.420552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.420552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.422397 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 998/1985] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:42:52 2022] DELETE /identity/v3/projects/b6c4100c21c44826a0c550533303c267 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:42:52.427853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b62116-d2a7-4f17-afca-62bdb32de75f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:52.440165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b62116-d2a7-4f17-afca-62bdb32de75f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:52.517438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:52.518085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:52.518360 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:52.518882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] PATH_INFO: `/v3/users/089441a372bd4b7a8ed7aa55cdb30df6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:52.520114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=089441a372bd4b7a8ed7aa55cdb30df6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:52.522357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:52.590298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '089441a372bd4b7a8ed7aa55cdb30df6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.599666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '089441a372bd4b7a8ed7aa55cdb30df6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.599988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '089441a372bd4b7a8ed7aa55cdb30df6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.600685 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '089441a372bd4b7a8ed7aa55cdb30df6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.612811 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0143a901-05e6-461f-a289-247f75678c0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '089441a372bd4b7a8ed7aa55cdb30df6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:52.645592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 988/1986] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:42:52 2022] DELETE /identity/v3/users/089441a372bd4b7a8ed7aa55cdb30df6 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:53.112189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:53.116528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:53.147248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea12840e-595c-4c27-94ab-631a157680d1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:53.249469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:53.250137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.250604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.250896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] PATH_INFO: `/v3/projects/35d19bb82dbe4d84a8dc0a534a0328f1` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.252106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] RBAC: Authorizing `identity:delete_project(project_id=35d19bb82dbe4d84a8dc0a534a0328f1)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:53.254022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:53.321926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35d19bb82dbe4d84a8dc0a534a0328f1', 'request_id': 'req-703f691f-3f0e-4ed6-a998-c93ea956a60c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:53.330950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35d19bb82dbe4d84a8dc0a534a0328f1', 'request_id': 'req-703f691f-3f0e-4ed6-a998-c93ea956a60c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:53.339410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35d19bb82dbe4d84a8dc0a534a0328f1', 'request_id': 'req-703f691f-3f0e-4ed6-a998-c93ea956a60c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:53.353640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] The token cache is being invalidate because project 35d19bb82dbe4d84a8dc0a534a0328f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:42:53.353862 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:53.354502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-703f691f-3f0e-4ed6-a998-c93ea956a60c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:53.356148 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 999/1987] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:42:53 2022] DELETE /identity/v3/projects/35d19bb82dbe4d84a8dc0a534a0328f1 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:53.361238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b001eda9-1f81-4d05-9564-6736744ed15f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.361521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b001eda9-1f81-4d05-9564-6736744ed15f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.361822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b001eda9-1f81-4d05-9564-6736744ed15f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.385375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-b001eda9-1f81-4d05-9564-6736744ed15f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:53.386702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b001eda9-1f81-4d05-9564-6736744ed15f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:53.426150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:53.468334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 989/1988] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:42:53.476181 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c3189a-e249-42fe-9755-5c733e73ae0e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.476463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c3189a-e249-42fe-9755-5c733e73ae0e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.476946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c3189a-e249-42fe-9755-5c733e73ae0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.500537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e9c3189a-e249-42fe-9755-5c733e73ae0e None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:53.502238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e9c3189a-e249-42fe-9755-5c733e73ae0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:53.514000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1000/1989] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:53.521265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b001eda9-1f81-4d05-9564-6736744ed15f None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:53.535562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b001eda9-1f81-4d05-9564-6736744ed15f None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:53.549051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:53.549887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.550325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.550781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.551805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:42:53.552127 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:53.554181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:53.572863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 990/1990] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:42:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:42:53.573718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c3189a-e249-42fe-9755-5c733e73ae0e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:53.585638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c3189a-e249-42fe-9755-5c733e73ae0e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:53.592685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9b65e4-8ebe-4295-aaf4-1a5631821b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:53.593365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b65e4-8ebe-4295-aaf4-1a5631821b52 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.593982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b65e4-8ebe-4295-aaf4-1a5631821b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.593982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b65e4-8ebe-4295-aaf4-1a5631821b52 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.594788 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9b65e4-8ebe-4295-aaf4-1a5631821b52 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:53.597103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9b65e4-8ebe-4295-aaf4-1a5631821b52 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:53.610914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:53.631339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1001/1991] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:42:53 2022] POST /identity/v3/projects => generated 363 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:53.635747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:53.651650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c1817f-7e23-4174-b5ba-6d36d14b01b1 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:53.721349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0038c19e-b07d-45b8-b557-80630d9f92bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:53.722132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038c19e-b07d-45b8-b557-80630d9f92bb admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.722418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038c19e-b07d-45b8-b557-80630d9f92bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.722721 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0038c19e-b07d-45b8-b557-80630d9f92bb admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.723369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0038c19e-b07d-45b8-b557-80630d9f92bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:53.725322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0038c19e-b07d-45b8-b557-80630d9f92bb admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:53.760844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 991/1992] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:42:53 2022] POST /identity/v3/users => generated 282 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:42:53.763149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9b65e4-8ebe-4295-aaf4-1a5631821b52 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:53.774624 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9b65e4-8ebe-4295-aaf4-1a5631821b52 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:53.780920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed922d58-0169-4240-9ed0-cc12e16f0228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:53.781550 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed922d58-0169-4240-9ed0-cc12e16f0228 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.781840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed922d58-0169-4240-9ed0-cc12e16f0228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.782118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed922d58-0169-4240-9ed0-cc12e16f0228 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.782810 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed922d58-0169-4240-9ed0-cc12e16f0228 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:53.784456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed922d58-0169-4240-9ed0-cc12e16f0228 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:53.799639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1002/1993] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:53 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:53.803727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0038c19e-b07d-45b8-b557-80630d9f92bb admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:53.814506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0038c19e-b07d-45b8-b557-80630d9f92bb admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:53.819551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc90cca-4ad3-46ec-8767-c70fd9a02a1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:53.820212 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc90cca-4ad3-46ec-8767-c70fd9a02a1b admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.820487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc90cca-4ad3-46ec-8767-c70fd9a02a1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.820796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc90cca-4ad3-46ec-8767-c70fd9a02a1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.821450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc90cca-4ad3-46ec-8767-c70fd9a02a1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:53.823104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc90cca-4ad3-46ec-8767-c70fd9a02a1b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:53.834787 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 992/1994] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:53.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed922d58-0169-4240-9ed0-cc12e16f0228 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:53.851551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed922d58-0169-4240-9ed0-cc12e16f0228 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:53.856492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f123e76-06d4-4f72-b017-22fe93896c36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:53.857784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f123e76-06d4-4f72-b017-22fe93896c36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.858071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f123e76-06d4-4f72-b017-22fe93896c36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.858350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f123e76-06d4-4f72-b017-22fe93896c36 admin admin] PATH_INFO: `/v3/projects/1996f323faa14fe2b552eea88933de5c/users/e651edde5a274fc588fe0184bee8a981/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.877236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f123e76-06d4-4f72-b017-22fe93896c36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1996f323faa14fe2b552eea88933de5c, user_id=e651edde5a274fc588fe0184bee8a981, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:53.879922 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f123e76-06d4-4f72-b017-22fe93896c36 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:53.894371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1003/1995] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:53 2022] PUT /identity/v3/projects/1996f323faa14fe2b552eea88933de5c/users/e651edde5a274fc588fe0184bee8a981/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:53.906247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042f642-a946-499a-97fd-c26885b9fe81 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:53.906575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042f642-a946-499a-97fd-c26885b9fe81 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:53.906848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042f642-a946-499a-97fd-c26885b9fe81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:53.944191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a042f642-a946-499a-97fd-c26885b9fe81 None None] MFA Rules not processed for user `e651edde5a274fc588fe0184bee8a981`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:53.945406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a042f642-a946-499a-97fd-c26885b9fe81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:54.019106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 993/1996] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:53 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:54.024786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f123e76-06d4-4f72-b017-22fe93896c36 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:54.105601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f123e76-06d4-4f72-b017-22fe93896c36 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:54.110950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2e4bc1-b907-46a4-bbd5-c2e00e80403e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:54.111655 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e4bc1-b907-46a4-bbd5-c2e00e80403e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:54.111932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e4bc1-b907-46a4-bbd5-c2e00e80403e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:54.112202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e4bc1-b907-46a4-bbd5-c2e00e80403e admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:54.112914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2e4bc1-b907-46a4-bbd5-c2e00e80403e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:54.114853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2e4bc1-b907-46a4-bbd5-c2e00e80403e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:54.149297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1004/1997] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:42:54 2022] POST /identity/v3/projects => generated 363 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:54.153338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a042f642-a946-499a-97fd-c26885b9fe81 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:54.165512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a042f642-a946-499a-97fd-c26885b9fe81 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:54.196195 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:54.235735 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5fbd01-ed75-4fc6-a173-ff4606f60a6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:54.236594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5fbd01-ed75-4fc6-a173-ff4606f60a6a admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:54.237078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5fbd01-ed75-4fc6-a173-ff4606f60a6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:54.237555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5fbd01-ed75-4fc6-a173-ff4606f60a6a admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:54.238634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5fbd01-ed75-4fc6-a173-ff4606f60a6a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:54.242295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5fbd01-ed75-4fc6-a173-ff4606f60a6a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:54.274589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 994/1998] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:42:54 2022] POST /identity/v3/users => generated 288 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:54.279745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2e4bc1-b907-46a4-bbd5-c2e00e80403e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:54.296035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2e4bc1-b907-46a4-bbd5-c2e00e80403e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:54.302656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7b19a9-e3d2-49dd-8fe3-2b5b84ad77fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:54.303959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7b19a9-e3d2-49dd-8fe3-2b5b84ad77fe admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:54.304631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7b19a9-e3d2-49dd-8fe3-2b5b84ad77fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:54.305105 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7b19a9-e3d2-49dd-8fe3-2b5b84ad77fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:54.306135 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7b19a9-e3d2-49dd-8fe3-2b5b84ad77fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:54.308235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7b19a9-e3d2-49dd-8fe3-2b5b84ad77fe admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:54.327144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1005/1999] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:54 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:42:54.334789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5fbd01-ed75-4fc6-a173-ff4606f60a6a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:54.347056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5fbd01-ed75-4fc6-a173-ff4606f60a6a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:54.352226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ad82d-258e-4ea4-a01b-19fe00c1d087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:54.353210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ad82d-258e-4ea4-a01b-19fe00c1d087 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:54.353210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ad82d-258e-4ea4-a01b-19fe00c1d087 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:54.353444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ad82d-258e-4ea4-a01b-19fe00c1d087 admin admin] PATH_INFO: `/v3/projects/efe589c1ce084e898ece873de8bd4b66/users/75e5c97838a743eb86a5a951a381a078/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:54.373291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487ad82d-258e-4ea4-a01b-19fe00c1d087 admin admin] RBAC: Authorizing `identity:create_grant(project_id=efe589c1ce084e898ece873de8bd4b66, user_id=75e5c97838a743eb86a5a951a381a078, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:54.378821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487ad82d-258e-4ea4-a01b-19fe00c1d087 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:54.404310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 995/2000] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:54 2022] PUT /identity/v3/projects/efe589c1ce084e898ece873de8bd4b66/users/75e5c97838a743eb86a5a951a381a078/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:54.412309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7b19a9-e3d2-49dd-8fe3-2b5b84ad77fe admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:54.440979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7b19a9-e3d2-49dd-8fe3-2b5b84ad77fe admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:54.593341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093e2f1-dde8-4dd4-b0bf-d5cad7b31fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:54.594370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093e2f1-dde8-4dd4-b0bf-d5cad7b31fc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:54.594684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093e2f1-dde8-4dd4-b0bf-d5cad7b31fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:54.594956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093e2f1-dde8-4dd4-b0bf-d5cad7b31fc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:54.595900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093e2f1-dde8-4dd4-b0bf-d5cad7b31fc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:54.597546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093e2f1-dde8-4dd4-b0bf-d5cad7b31fc0 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:54.621073 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1006/2001] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:54 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:54.626512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ad82d-258e-4ea4-a01b-19fe00c1d087 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:54.641075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ad82d-258e-4ea4-a01b-19fe00c1d087 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:54.648494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d65619-7c1d-452d-a76f-289043e594da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:54.649529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d65619-7c1d-452d-a76f-289043e594da admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:54.650213 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d65619-7c1d-452d-a76f-289043e594da admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:54.650748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d65619-7c1d-452d-a76f-289043e594da admin admin] PATH_INFO: `/v3/projects/efe589c1ce084e898ece873de8bd4b66/users/75e5c97838a743eb86a5a951a381a078/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:54.654468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d65619-7c1d-452d-a76f-289043e594da admin admin] RBAC: Authorizing `identity:create_grant(project_id=efe589c1ce084e898ece873de8bd4b66, user_id=75e5c97838a743eb86a5a951a381a078, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:54.660802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d65619-7c1d-452d-a76f-289043e594da admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:54.683535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 996/2002] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:54 2022] PUT /identity/v3/projects/efe589c1ce084e898ece873de8bd4b66/users/75e5c97838a743eb86a5a951a381a078/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:54.695524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e57829-7984-4413-a6d2-4a3ec34b2dc3 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:54.695814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e57829-7984-4413-a6d2-4a3ec34b2dc3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:54.696095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e57829-7984-4413-a6d2-4a3ec34b2dc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:54.742518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-42e57829-7984-4413-a6d2-4a3ec34b2dc3 None None] MFA Rules not processed for user `75e5c97838a743eb86a5a951a381a078`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:54.744336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-42e57829-7984-4413-a6d2-4a3ec34b2dc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:54.791336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:54.913296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7787ccb-b83a-43a2-a8e0-e8a90374f02c None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:54.913654 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7787ccb-b83a-43a2-a8e0-e8a90374f02c None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:54.913865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7787ccb-b83a-43a2-a8e0-e8a90374f02c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:54.943768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e7787ccb-b83a-43a2-a8e0-e8a90374f02c None None] MFA Rules not processed for user `39de6a6fff7c4cc08cc62aa86182d71d`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:54.948361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e7787ccb-b83a-43a2-a8e0-e8a90374f02c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:54.957890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1007/2003] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:54 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:54.973704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e57829-7984-4413-a6d2-4a3ec34b2dc3 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:54.977140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-42e57829-7984-4413-a6d2-4a3ec34b2dc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:55.122004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e57829-7984-4413-a6d2-4a3ec34b2dc3 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:55.131915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08778760-8158-4874-a681-b70152b6fd1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:55.134739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08778760-8158-4874-a681-b70152b6fd1a service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:55.135638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08778760-8158-4874-a681-b70152b6fd1a service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:55.136466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08778760-8158-4874-a681-b70152b6fd1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:55.147923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 997/2004] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:54 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:55.155642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08778760-8158-4874-a681-b70152b6fd1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:55.159012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08778760-8158-4874-a681-b70152b6fd1a service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:55.186734 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1008/2005] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:54 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:55.744795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7787ccb-b83a-43a2-a8e0-e8a90374f02c None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:55.845978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7787ccb-b83a-43a2-a8e0-e8a90374f02c None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:55.852522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:55.853176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:55.853459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:55.853736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] PATH_INFO: `/v3/users/e651edde5a274fc588fe0184bee8a981` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:55.855015 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e651edde5a274fc588fe0184bee8a981)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:55.857860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:55.931365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:55.965196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e651edde5a274fc588fe0184bee8a981'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:55.985328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e651edde5a274fc588fe0184bee8a981'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:55.986137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e651edde5a274fc588fe0184bee8a981'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:55.987014 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e651edde5a274fc588fe0184bee8a981'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:55.997923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e651edde5a274fc588fe0184bee8a981'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:56.045299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 998/2006] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:42:55 2022] DELETE /identity/v3/users/e651edde5a274fc588fe0184bee8a981 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:56.049545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984a8d16-fbf2-4c71-8482-dc77a38090e8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:56.050063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984a8d16-fbf2-4c71-8482-dc77a38090e8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:56.051016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984a8d16-fbf2-4c71-8482-dc77a38090e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:56.117742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-984a8d16-fbf2-4c71-8482-dc77a38090e8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:56.121171 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-984a8d16-fbf2-4c71-8482-dc77a38090e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:56.214097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1009/2007] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:42:56.222086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:56.223782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:56.300256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d5e402-515e-459c-af9c-452192c71753 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:56.305057 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ad4680-e2a9-4202-bda9-0b9b6c68e1aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:56.305710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ad4680-e2a9-4202-bda9-0b9b6c68e1aa service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:56.305994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ad4680-e2a9-4202-bda9-0b9b6c68e1aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:56.306388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ad4680-e2a9-4202-bda9-0b9b6c68e1aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:56.323146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ad4680-e2a9-4202-bda9-0b9b6c68e1aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:56.324903 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ad4680-e2a9-4202-bda9-0b9b6c68e1aa service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:56.347870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 999/2008] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:42:57.430152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984a8d16-fbf2-4c71-8482-dc77a38090e8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:57.431734 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-984a8d16-fbf2-4c71-8482-dc77a38090e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:57.454208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984a8d16-fbf2-4c71-8482-dc77a38090e8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:57.475284 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:57.475952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:57.476237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:57.476518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] PATH_INFO: `/v3/projects/1996f323faa14fe2b552eea88933de5c` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:57.478374 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] RBAC: Authorizing `identity:delete_project(project_id=1996f323faa14fe2b552eea88933de5c)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:57.490982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:57.626830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1996f323faa14fe2b552eea88933de5c', 'request_id': 'req-a33f5e98-8538-4cd6-8304-849de13ef62c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.634978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:57.650364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1996f323faa14fe2b552eea88933de5c', 'request_id': 'req-a33f5e98-8538-4cd6-8304-849de13ef62c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.663771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1996f323faa14fe2b552eea88933de5c', 'request_id': 'req-a33f5e98-8538-4cd6-8304-849de13ef62c'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.687924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] The token cache is being invalidate because project 1996f323faa14fe2b552eea88933de5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:42:57.688230 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.691494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.693901 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1010/2009] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:42:57 2022] DELETE /identity/v3/projects/1996f323faa14fe2b552eea88933de5c => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:57.698122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ad4680-e2a9-4202-bda9-0b9b6c68e1aa service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:57.714405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ad4680-e2a9-4202-bda9-0b9b6c68e1aa service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:57.816907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:57.817712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:57.817995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:57.818318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] PATH_INFO: `/v3/users/75e5c97838a743eb86a5a951a381a078` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:57.819687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75e5c97838a743eb86a5a951a381a078)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:57.822879 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:57.889097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75e5c97838a743eb86a5a951a381a078'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.896643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:57.899156 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75e5c97838a743eb86a5a951a381a078'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.901325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75e5c97838a743eb86a5a951a381a078'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.901487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75e5c97838a743eb86a5a951a381a078'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.912514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-71bf878b-c54b-4c61-870a-99ee357c1c92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75e5c97838a743eb86a5a951a381a078'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:57.932104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1000/2010] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:42:57 2022] DELETE /identity/v3/users/75e5c97838a743eb86a5a951a381a078 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:58.273731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:58.276043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:58.292959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33f5e98-8538-4cd6-8304-849de13ef62c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:58.363878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:58.364853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:58.364853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:58.365052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] PATH_INFO: `/v3/projects/efe589c1ce084e898ece873de8bd4b66` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:58.366270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] RBAC: Authorizing `identity:delete_project(project_id=efe589c1ce084e898ece873de8bd4b66)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:58.368479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:58.436681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efe589c1ce084e898ece873de8bd4b66', 'request_id': 'req-78591e58-26fe-432d-9da5-a3ce2aa3253b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:58.444354 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efe589c1ce084e898ece873de8bd4b66', 'request_id': 'req-78591e58-26fe-432d-9da5-a3ce2aa3253b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:58.458981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efe589c1ce084e898ece873de8bd4b66', 'request_id': 'req-78591e58-26fe-432d-9da5-a3ce2aa3253b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:58.498962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] The token cache is being invalidate because project efe589c1ce084e898ece873de8bd4b66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:42:58.499410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:58.500823 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78591e58-26fe-432d-9da5-a3ce2aa3253b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:42:58.503054 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1011/2011] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:42:58 2022] DELETE /identity/v3/projects/efe589c1ce084e898ece873de8bd4b66 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:58.509078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ed871f-d7e9-461a-972c-34fcd5145fe3 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:58.510032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ed871f-d7e9-461a-972c-34fcd5145fe3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:58.510300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ed871f-d7e9-461a-972c-34fcd5145fe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:58.537579 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-88ed871f-d7e9-461a-972c-34fcd5145fe3 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:58.539151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-88ed871f-d7e9-461a-972c-34fcd5145fe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:58.619013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1001/2012] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:58.627888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc8ccd8-13f6-4250-8c37-9cdf1708846d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:58.628302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc8ccd8-13f6-4250-8c37-9cdf1708846d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:58.628587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc8ccd8-13f6-4250-8c37-9cdf1708846d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:58.652130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-6bc8ccd8-13f6-4250-8c37-9cdf1708846d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:58.653400 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6bc8ccd8-13f6-4250-8c37-9cdf1708846d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:58.663942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1012/2013] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:42:58.668344 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ed871f-d7e9-461a-972c-34fcd5145fe3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:58.679287 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ed871f-d7e9-461a-972c-34fcd5145fe3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:58.685229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:58.685896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:58.686181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:58.686457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:58.687187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:42:58.687499 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:58.689117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:58.703187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1002/2014] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:42:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:42:58.707829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc8ccd8-13f6-4250-8c37-9cdf1708846d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:58.721000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc8ccd8-13f6-4250-8c37-9cdf1708846d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:58.727347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db44dd3e-3027-42c4-bba3-8579422ddc38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:58.728164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db44dd3e-3027-42c4-bba3-8579422ddc38 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:58.728668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db44dd3e-3027-42c4-bba3-8579422ddc38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:58.729399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db44dd3e-3027-42c4-bba3-8579422ddc38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:58.730410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db44dd3e-3027-42c4-bba3-8579422ddc38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:58.733314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db44dd3e-3027-42c4-bba3-8579422ddc38 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:58.768356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1013/2015] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:42:58 2022] POST /identity/v3/projects => generated 365 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:42:58.773674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:58.789822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d424bcb-01c9-49bc-a0bd-74461c6862d9 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:58.869099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0438c1-69a9-4952-bf0e-ccb58d24303f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:58.869715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0438c1-69a9-4952-bf0e-ccb58d24303f admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:58.869996 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0438c1-69a9-4952-bf0e-ccb58d24303f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:58.870270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0438c1-69a9-4952-bf0e-ccb58d24303f admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:58.871045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0438c1-69a9-4952-bf0e-ccb58d24303f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:58.873075 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0438c1-69a9-4952-bf0e-ccb58d24303f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:58.897385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1003/2016] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:42:58 2022] POST /identity/v3/users => generated 283 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:58.902407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db44dd3e-3027-42c4-bba3-8579422ddc38 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:58.919334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db44dd3e-3027-42c4-bba3-8579422ddc38 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:58.925670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf75aa92-1732-4428-9342-dc00e05aecef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:58.926300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf75aa92-1732-4428-9342-dc00e05aecef admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:58.926616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf75aa92-1732-4428-9342-dc00e05aecef admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf75aa92-1732-4428-9342-dc00e05aecef admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:58.927555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf75aa92-1732-4428-9342-dc00e05aecef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:58.929163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf75aa92-1732-4428-9342-dc00e05aecef admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:58.941282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1014/2017] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:58 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:58.945386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0438c1-69a9-4952-bf0e-ccb58d24303f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:58.956188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0438c1-69a9-4952-bf0e-ccb58d24303f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:58.961030 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972698f4-d752-4251-b583-80fdcc465ca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:58.961636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972698f4-d752-4251-b583-80fdcc465ca1 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:58.961908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972698f4-d752-4251-b583-80fdcc465ca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:58.962175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972698f4-d752-4251-b583-80fdcc465ca1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:58.962952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972698f4-d752-4251-b583-80fdcc465ca1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:58.964552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972698f4-d752-4251-b583-80fdcc465ca1 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:58.978652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1004/2018] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:58.981912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf75aa92-1732-4428-9342-dc00e05aecef admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:58.996356 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf75aa92-1732-4428-9342-dc00e05aecef admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:59.001317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4299d83-c817-4dc2-9232-6199df37c4a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:59.001993 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4299d83-c817-4dc2-9232-6199df37c4a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.002276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4299d83-c817-4dc2-9232-6199df37c4a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.002590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4299d83-c817-4dc2-9232-6199df37c4a5 admin admin] PATH_INFO: `/v3/projects/d7d5fbf0aa864788be5388af0234da94/users/766c40714c8d44d39a8db344809e665c/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.021602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4299d83-c817-4dc2-9232-6199df37c4a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7d5fbf0aa864788be5388af0234da94, user_id=766c40714c8d44d39a8db344809e665c, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:59.024547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4299d83-c817-4dc2-9232-6199df37c4a5 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:59.042421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1015/2019] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:58 2022] PUT /identity/v3/projects/d7d5fbf0aa864788be5388af0234da94/users/766c40714c8d44d39a8db344809e665c/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:59.054126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33def30a-74b1-4266-b804-bc62ec0bc2f4 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.054464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33def30a-74b1-4266-b804-bc62ec0bc2f4 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.054771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33def30a-74b1-4266-b804-bc62ec0bc2f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.099333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-33def30a-74b1-4266-b804-bc62ec0bc2f4 None None] MFA Rules not processed for user `766c40714c8d44d39a8db344809e665c`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:59.105319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-33def30a-74b1-4266-b804-bc62ec0bc2f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:59.136279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:59.183397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1005/2020] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:59 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:59.188857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4299d83-c817-4dc2-9232-6199df37c4a5 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:59.194721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:59.257511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4299d83-c817-4dc2-9232-6199df37c4a5 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:59.262199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6782a206-d52a-4310-9a3c-a67fde0e3c85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:59.262847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6782a206-d52a-4310-9a3c-a67fde0e3c85 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.263125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6782a206-d52a-4310-9a3c-a67fde0e3c85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.263970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6782a206-d52a-4310-9a3c-a67fde0e3c85 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.264616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6782a206-d52a-4310-9a3c-a67fde0e3c85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:59.266537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6782a206-d52a-4310-9a3c-a67fde0e3c85 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:59.293931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1016/2021] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:42:59 2022] POST /identity/v3/projects => generated 365 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:42:59.298242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33def30a-74b1-4266-b804-bc62ec0bc2f4 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:59.309577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33def30a-74b1-4266-b804-bc62ec0bc2f4 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:59.379196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0514eb40-b8de-4a91-9b74-e1c92f2f310c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:59.379914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514eb40-b8de-4a91-9b74-e1c92f2f310c admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.380273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514eb40-b8de-4a91-9b74-e1c92f2f310c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.380625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514eb40-b8de-4a91-9b74-e1c92f2f310c admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.381354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0514eb40-b8de-4a91-9b74-e1c92f2f310c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:59.383479 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0514eb40-b8de-4a91-9b74-e1c92f2f310c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:59.421754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1006/2022] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:42:59 2022] POST /identity/v3/users => generated 289 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:42:59.423084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6782a206-d52a-4310-9a3c-a67fde0e3c85 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:59.434430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6782a206-d52a-4310-9a3c-a67fde0e3c85 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:59.439338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936c520-0fcd-40bd-bf3b-bca6ae7a772b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:59.440035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936c520-0fcd-40bd-bf3b-bca6ae7a772b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.440449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936c520-0fcd-40bd-bf3b-bca6ae7a772b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.440589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936c520-0fcd-40bd-bf3b-bca6ae7a772b admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.441227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3936c520-0fcd-40bd-bf3b-bca6ae7a772b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:59.442888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3936c520-0fcd-40bd-bf3b-bca6ae7a772b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:59.458370 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1017/2023] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:59.459673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0514eb40-b8de-4a91-9b74-e1c92f2f310c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:59.471727 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0514eb40-b8de-4a91-9b74-e1c92f2f310c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:59.476573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdbac56-ca96-4cf1-acaa-aeb6492f3ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:59.477285 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdbac56-ca96-4cf1-acaa-aeb6492f3ad7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.477565 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdbac56-ca96-4cf1-acaa-aeb6492f3ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.477841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdbac56-ca96-4cf1-acaa-aeb6492f3ad7 admin admin] PATH_INFO: `/v3/projects/156e965bcd47408897c186cc333a2e9d/users/e1f3a3b78cd1487aaf873e187b332e80/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.497656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdbac56-ca96-4cf1-acaa-aeb6492f3ad7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=156e965bcd47408897c186cc333a2e9d, user_id=e1f3a3b78cd1487aaf873e187b332e80, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:59.500476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdbac56-ca96-4cf1-acaa-aeb6492f3ad7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:59.522040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1007/2024] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:59 2022] PUT /identity/v3/projects/156e965bcd47408897c186cc333a2e9d/users/e1f3a3b78cd1487aaf873e187b332e80/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:59.526114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936c520-0fcd-40bd-bf3b-bca6ae7a772b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:59.537419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936c520-0fcd-40bd-bf3b-bca6ae7a772b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:59.609771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32c633c-6536-4e9a-b37c-2378fcb4a6b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:59.610393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c633c-6536-4e9a-b37c-2378fcb4a6b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.610699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c633c-6536-4e9a-b37c-2378fcb4a6b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.610996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c633c-6536-4e9a-b37c-2378fcb4a6b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.611641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32c633c-6536-4e9a-b37c-2378fcb4a6b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:59.613238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32c633c-6536-4e9a-b37c-2378fcb4a6b8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:59.624365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1018/2025] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:42:59 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:42:59.628651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdbac56-ca96-4cf1-acaa-aeb6492f3ad7 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:59.639907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdbac56-ca96-4cf1-acaa-aeb6492f3ad7 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:59.644645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:59.645313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.645595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.645872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] PATH_INFO: `/v3/projects/156e965bcd47408897c186cc333a2e9d/users/e1f3a3b78cd1487aaf873e187b332e80/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.647836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=156e965bcd47408897c186cc333a2e9d, user_id=e1f3a3b78cd1487aaf873e187b332e80, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:59.650477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:59.661941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1008/2026] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:42:59 2022] PUT /identity/v3/projects/156e965bcd47408897c186cc333a2e9d/users/e1f3a3b78cd1487aaf873e187b332e80/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:42:59.673629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd610494-956c-48d0-917f-945640f4cefd None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.674019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd610494-956c-48d0-917f-945640f4cefd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.674308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd610494-956c-48d0-917f-945640f4cefd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.703570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-cd610494-956c-48d0-917f-945640f4cefd None None] MFA Rules not processed for user `e1f3a3b78cd1487aaf873e187b332e80`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:42:59.704813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cd610494-956c-48d0-917f-945640f4cefd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:59.780143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:42:59.785445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1019/2027] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:42:59 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:42:59.797066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:42:59.799478 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:42:59.866980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480bb692-5a63-4f1b-8a46-2f2d0d0302b1 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:42:59.873129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb9a499-53a2-4fe9-976a-0837caed4105 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:42:59.874432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9a499-53a2-4fe9-976a-0837caed4105 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:42:59.875364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9a499-53a2-4fe9-976a-0837caed4105 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:42:59.875778 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9a499-53a2-4fe9-976a-0837caed4105 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:42:59.887201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb9a499-53a2-4fe9-976a-0837caed4105 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:42:59.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb9a499-53a2-4fe9-976a-0837caed4105 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:42:59.899885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1009/2028] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:59 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:42:59.997294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd610494-956c-48d0-917f-945640f4cefd None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:00.007642 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd610494-956c-48d0-917f-945640f4cefd None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:00.011651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:00.012303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:00.012670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:00.012950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:00.080410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:00.081916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:00.092442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1020/2029] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:42:59 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:03.092474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb24fe3-c324-4de1-8613-0fb76198818d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:03.093000 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb24fe3-c324-4de1-8613-0fb76198818d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:03.093175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb24fe3-c324-4de1-8613-0fb76198818d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:03.125255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ceb24fe3-c324-4de1-8613-0fb76198818d None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:03.128097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ceb24fe3-c324-4de1-8613-0fb76198818d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:03.161428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:03.238206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1010/2030] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:43:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:43:03.293979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:03.296635 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:03.403023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791c2e17-4823-49b2-9ff3-1aa682677055 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:03.410254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d014a937-f4b8-4a09-b4e8-dffae57b1044 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:03.411207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014a937-f4b8-4a09-b4e8-dffae57b1044 service swift] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:03.411607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014a937-f4b8-4a09-b4e8-dffae57b1044 service swift] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:03.412169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014a937-f4b8-4a09-b4e8-dffae57b1044 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:03.435873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d014a937-f4b8-4a09-b4e8-dffae57b1044 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:03.444826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d014a937-f4b8-4a09-b4e8-dffae57b1044 service swift] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:03.491885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1021/2031] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:43:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:43:09.186169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb24fe3-c324-4de1-8613-0fb76198818d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:09.187805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ceb24fe3-c324-4de1-8613-0fb76198818d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:09.333642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb24fe3-c324-4de1-8613-0fb76198818d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:09.354854 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:09.355932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:09.355932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:09.356143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] PATH_INFO: `/v3/users/9b938d369eda43bf9258c9103676e5b7` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:09.358153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b938d369eda43bf9258c9103676e5b7)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:09.360412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:09.443004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b938d369eda43bf9258c9103676e5b7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:09.452770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b938d369eda43bf9258c9103676e5b7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:09.453546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b938d369eda43bf9258c9103676e5b7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:09.454343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b938d369eda43bf9258c9103676e5b7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:09.468222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b938d369eda43bf9258c9103676e5b7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:09.527576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1011/2032] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:43:09 2022] DELETE /identity/v3/users/9b938d369eda43bf9258c9103676e5b7 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:43:09.531636 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b31835-2cf0-4740-8c46-438616e13fa8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:09.532323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b31835-2cf0-4740-8c46-438616e13fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:09.532323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b31835-2cf0-4740-8c46-438616e13fa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:09.616410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-18b31835-2cf0-4740-8c46-438616e13fa8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:09.620011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-18b31835-2cf0-4740-8c46-438616e13fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:09.825038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1022/2033] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:09.986943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:09.989016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:10.206916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:10.227101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e101d2-de5c-4b1c-814b-bb5198a922cd admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:10.238247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:10.239551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:10.240164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:10.240789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:10.270110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:10.273257 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:10.303136 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1012/2034] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:43:11.576358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b31835-2cf0-4740-8c46-438616e13fa8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:11.581040 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-18b31835-2cf0-4740-8c46-438616e13fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:11.631975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b31835-2cf0-4740-8c46-438616e13fa8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:11.643103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:11.644669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:11.645177 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:11.645671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] PATH_INFO: `/v3/projects/b403ad5109174ed0967e2c657f3298b9` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:11.647646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] RBAC: Authorizing `identity:delete_project(project_id=b403ad5109174ed0967e2c657f3298b9)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:11.651035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:11.701725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:11.771253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b403ad5109174ed0967e2c657f3298b9', 'request_id': 'req-24daca7c-158b-49e6-855c-d1a33e7bc8db'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:11.789896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b403ad5109174ed0967e2c657f3298b9', 'request_id': 'req-24daca7c-158b-49e6-855c-d1a33e7bc8db'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:11.809711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b403ad5109174ed0967e2c657f3298b9', 'request_id': 'req-24daca7c-158b-49e6-855c-d1a33e7bc8db'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:11.838614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] The token cache is being invalidate because project b403ad5109174ed0967e2c657f3298b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:43:11.838926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:11.840396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-24daca7c-158b-49e6-855c-d1a33e7bc8db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:11.845512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1023/2035] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:43:11 2022] DELETE /identity/v3/projects/b403ad5109174ed0967e2c657f3298b9 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:13.341862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:13.343466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:13.357273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a18d3f-3651-4cf7-b488-32d05dea27e8 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:13.434091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:13.435452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:13.435828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:13.436169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] PATH_INFO: `/v3/users/766c40714c8d44d39a8db344809e665c` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:13.437362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] RBAC: Authorizing `identity:delete_user(user_id=766c40714c8d44d39a8db344809e665c)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:13.440668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:13.512339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '766c40714c8d44d39a8db344809e665c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:13.525320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '766c40714c8d44d39a8db344809e665c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:13.526607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '766c40714c8d44d39a8db344809e665c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:13.527598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '766c40714c8d44d39a8db344809e665c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:13.539548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '766c40714c8d44d39a8db344809e665c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:13.578821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1013/2036] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:43:13 2022] DELETE /identity/v3/users/766c40714c8d44d39a8db344809e665c => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:13.580640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612757ef-b830-45a5-b6fe-d6bc250866ab None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:13.581142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612757ef-b830-45a5-b6fe-d6bc250866ab None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:13.581259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612757ef-b830-45a5-b6fe-d6bc250866ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:13.605288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-612757ef-b830-45a5-b6fe-d6bc250866ab None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:13.607215 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-612757ef-b830-45a5-b6fe-d6bc250866ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:13.723289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1024/2037] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:13.729073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:13.731816 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:13.808702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fe6705-9f0c-4478-8c41-3978395da55e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:13.813907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473ced5d-5a5f-4aae-9d80-3a3029f65e2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:13.814948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ced5d-5a5f-4aae-9d80-3a3029f65e2a service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:13.815403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ced5d-5a5f-4aae-9d80-3a3029f65e2a service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:13.815848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473ced5d-5a5f-4aae-9d80-3a3029f65e2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:13.831299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473ced5d-5a5f-4aae-9d80-3a3029f65e2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:13.833264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473ced5d-5a5f-4aae-9d80-3a3029f65e2a service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:13.847512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1014/2038] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:14.241556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612757ef-b830-45a5-b6fe-d6bc250866ab None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:14.244565 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-612757ef-b830-45a5-b6fe-d6bc250866ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:14.281038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612757ef-b830-45a5-b6fe-d6bc250866ab None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:14.310568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:14.313390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:14.313390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:14.313390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] PATH_INFO: `/v3/projects/d7d5fbf0aa864788be5388af0234da94` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:14.313390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7d5fbf0aa864788be5388af0234da94)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:14.315128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:14.411721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7d5fbf0aa864788be5388af0234da94', 'request_id': 'req-32f13107-d6b7-49ca-9092-89b12efa587e'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.418659 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7d5fbf0aa864788be5388af0234da94', 'request_id': 'req-32f13107-d6b7-49ca-9092-89b12efa587e'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.429745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7d5fbf0aa864788be5388af0234da94', 'request_id': 'req-32f13107-d6b7-49ca-9092-89b12efa587e'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.451185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] The token cache is being invalidate because project d7d5fbf0aa864788be5388af0234da94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:43:14.451958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.452766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.454964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1025/2039] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:43:14 2022] DELETE /identity/v3/projects/d7d5fbf0aa864788be5388af0234da94 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:14.458939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473ced5d-5a5f-4aae-9d80-3a3029f65e2a service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:14.473034 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473ced5d-5a5f-4aae-9d80-3a3029f65e2a service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:14.483230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:14.567929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:14.568684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:14.569036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:14.569379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] PATH_INFO: `/v3/users/e1f3a3b78cd1487aaf873e187b332e80` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:14.570689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1f3a3b78cd1487aaf873e187b332e80)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:14.572935 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:14.630767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1f3a3b78cd1487aaf873e187b332e80'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.639577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1f3a3b78cd1487aaf873e187b332e80'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.640629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1f3a3b78cd1487aaf873e187b332e80'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.641310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1f3a3b78cd1487aaf873e187b332e80'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.651361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-88da63a5-8c83-4ff3-8614-e0ff24c969ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1f3a3b78cd1487aaf873e187b332e80'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:14.672673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1015/2040] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:43:14 2022] DELETE /identity/v3/users/e1f3a3b78cd1487aaf873e187b332e80 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:15.128039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:15.133865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:15.158691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f13107-d6b7-49ca-9092-89b12efa587e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:15.246695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:15.289807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:15.290696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:15.291065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:15.291411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] PATH_INFO: `/v3/projects/156e965bcd47408897c186cc333a2e9d` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:15.292774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] RBAC: Authorizing `identity:delete_project(project_id=156e965bcd47408897c186cc333a2e9d)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:15.296133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:15.365463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '156e965bcd47408897c186cc333a2e9d', 'request_id': 'req-52e7c1f2-6f35-41fe-8c2d-9e0336271940'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:15.382093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '156e965bcd47408897c186cc333a2e9d', 'request_id': 'req-52e7c1f2-6f35-41fe-8c2d-9e0336271940'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:15.394842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '156e965bcd47408897c186cc333a2e9d', 'request_id': 'req-52e7c1f2-6f35-41fe-8c2d-9e0336271940'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:15.409192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] The token cache is being invalidate because project 156e965bcd47408897c186cc333a2e9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:43:15.409755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:15.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-52e7c1f2-6f35-41fe-8c2d-9e0336271940 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:15.414417 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1026/2041] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:43:15 2022] DELETE /identity/v3/projects/156e965bcd47408897c186cc333a2e9d => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:15.420340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8075e-683c-4599-835a-3c50da66cb6d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:15.420678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8075e-683c-4599-835a-3c50da66cb6d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:15.420896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8075e-683c-4599-835a-3c50da66cb6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:15.445992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-44a8075e-683c-4599-835a-3c50da66cb6d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:15.447331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-44a8075e-683c-4599-835a-3c50da66cb6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:15.576652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1016/2042] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:15.577960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4900ecfa-14c3-4fbb-979c-2ac0309c25f4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:15.578404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4900ecfa-14c3-4fbb-979c-2ac0309c25f4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:15.578886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4900ecfa-14c3-4fbb-979c-2ac0309c25f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:15.622769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-4900ecfa-14c3-4fbb-979c-2ac0309c25f4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:15.625777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4900ecfa-14c3-4fbb-979c-2ac0309c25f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:15.656532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a8075e-683c-4599-835a-3c50da66cb6d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:15.657252 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1027/2043] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:15.668451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a8075e-683c-4599-835a-3c50da66cb6d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:15.677690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:15.678497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:15.678896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:15.679270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:15.680040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:43:15.681272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:15.683171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:15.705662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4900ecfa-14c3-4fbb-979c-2ac0309c25f4 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:15.712299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1017/2044] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:43:15.716890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4900ecfa-14c3-4fbb-979c-2ac0309c25f4 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:15.739312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9f9831-970f-4d07-9991-b5f1c5f82f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:15.740037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9f9831-970f-4d07-9991-b5f1c5f82f82 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:15.740315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9f9831-970f-4d07-9991-b5f1c5f82f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:15.740591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9f9831-970f-4d07-9991-b5f1c5f82f82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:15.741237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9f9831-970f-4d07-9991-b5f1c5f82f82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:15.745615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9f9831-970f-4d07-9991-b5f1c5f82f82 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:15.779513 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1028/2045] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:43:15 2022] POST /identity/v3/projects => generated 375 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:43:15.783012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:15.813795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee677a62-0de9-44f1-bdb7-de3d8c8a6286 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:15.826917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:15.883538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511148f-5d89-4eb7-87c4-805580db306f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:15.884311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511148f-5d89-4eb7-87c4-805580db306f admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:15.884601 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511148f-5d89-4eb7-87c4-805580db306f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:15.884875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511148f-5d89-4eb7-87c4-805580db306f admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:15.885512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e511148f-5d89-4eb7-87c4-805580db306f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:15.887474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e511148f-5d89-4eb7-87c4-805580db306f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:15.917566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1018/2046] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:43:15 2022] POST /identity/v3/users => generated 288 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:43:15.918903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9f9831-970f-4d07-9991-b5f1c5f82f82 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:15.930235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9f9831-970f-4d07-9991-b5f1c5f82f82 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:15.939127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab187b0-0fbd-463c-b07b-4b8bf51fd2d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:15.939775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab187b0-0fbd-463c-b07b-4b8bf51fd2d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:15.940047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab187b0-0fbd-463c-b07b-4b8bf51fd2d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:15.940318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab187b0-0fbd-463c-b07b-4b8bf51fd2d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:15.940962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab187b0-0fbd-463c-b07b-4b8bf51fd2d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:15.942631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab187b0-0fbd-463c-b07b-4b8bf51fd2d3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:15.955408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1029/2047] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:43:15 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:43:15.959184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511148f-5d89-4eb7-87c4-805580db306f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:15.970442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511148f-5d89-4eb7-87c4-805580db306f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:15.976582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c0a31f-807a-430a-8695-65440e2c960b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:15.977495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c0a31f-807a-430a-8695-65440e2c960b admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:15.977777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c0a31f-807a-430a-8695-65440e2c960b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:15.978051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c0a31f-807a-430a-8695-65440e2c960b admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:15.978776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c0a31f-807a-430a-8695-65440e2c960b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:15.980381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c0a31f-807a-430a-8695-65440e2c960b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:15.998003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1019/2048] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:43:15 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:43:15.999310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab187b0-0fbd-463c-b07b-4b8bf51fd2d3 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.011311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab187b0-0fbd-463c-b07b-4b8bf51fd2d3 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:16.017366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165842f9-8c68-4dbd-b0a9-35d088bb0fe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:16.019997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165842f9-8c68-4dbd-b0a9-35d088bb0fe4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.020489 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165842f9-8c68-4dbd-b0a9-35d088bb0fe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.020948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165842f9-8c68-4dbd-b0a9-35d088bb0fe4 admin admin] PATH_INFO: `/v3/projects/414577d72f964d759d66d26cf8f55a96/users/1dcc438968d64085a15be58c5a049718/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.039472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165842f9-8c68-4dbd-b0a9-35d088bb0fe4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=414577d72f964d759d66d26cf8f55a96, user_id=1dcc438968d64085a15be58c5a049718, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:16.042578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165842f9-8c68-4dbd-b0a9-35d088bb0fe4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:16.080049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7bec3c-8e32-418e-bf67-c834fe6640d8 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.080719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7bec3c-8e32-418e-bf67-c834fe6640d8 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.081294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7bec3c-8e32-418e-bf67-c834fe6640d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.091008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1030/2049] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:43:15 2022] PUT /identity/v3/projects/414577d72f964d759d66d26cf8f55a96/users/1dcc438968d64085a15be58c5a049718/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:16.140600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8c7bec3c-8e32-418e-bf67-c834fe6640d8 None None] MFA Rules not processed for user `1dcc438968d64085a15be58c5a049718`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:16.142140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8c7bec3c-8e32-418e-bf67-c834fe6640d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:16.237687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1020/2050] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:16 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:43:16.241979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165842f9-8c68-4dbd-b0a9-35d088bb0fe4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.321712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165842f9-8c68-4dbd-b0a9-35d088bb0fe4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:16.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55af67b6-d0e2-4632-8da9-385fbec5b50c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:16.336716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55af67b6-d0e2-4632-8da9-385fbec5b50c admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.337027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55af67b6-d0e2-4632-8da9-385fbec5b50c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.337540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55af67b6-d0e2-4632-8da9-385fbec5b50c admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.339009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55af67b6-d0e2-4632-8da9-385fbec5b50c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:16.343176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55af67b6-d0e2-4632-8da9-385fbec5b50c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:16.354793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:16.384122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc35a1a-5095-4800-a072-2e716165ef2a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.384378 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc35a1a-5095-4800-a072-2e716165ef2a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.384645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc35a1a-5095-4800-a072-2e716165ef2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.387253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1031/2051] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:43:16 2022] POST /identity/v3/projects => generated 377 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:43:16.392574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55af67b6-d0e2-4632-8da9-385fbec5b50c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.411044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-adc35a1a-5095-4800-a072-2e716165ef2a None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:16.412733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-adc35a1a-5095-4800-a072-2e716165ef2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:16.473891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55af67b6-d0e2-4632-8da9-385fbec5b50c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:16.484232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677d6d92-52f1-4783-bce0-e014ccdd4167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:16.485038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677d6d92-52f1-4783-bce0-e014ccdd4167 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.485779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677d6d92-52f1-4783-bce0-e014ccdd4167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.487038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677d6d92-52f1-4783-bce0-e014ccdd4167 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.487930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677d6d92-52f1-4783-bce0-e014ccdd4167 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:16.491773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677d6d92-52f1-4783-bce0-e014ccdd4167 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:16.495140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1021/2052] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:43:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:16.502174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc35a1a-5095-4800-a072-2e716165ef2a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-adc35a1a-5095-4800-a072-2e716165ef2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:16.522911 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1032/2053] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:43:16 2022] POST /identity/v3/users => generated 295 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:43:16.524020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677d6d92-52f1-4783-bce0-e014ccdd4167 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.534348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677d6d92-52f1-4783-bce0-e014ccdd4167 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:16.539131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24702a3-0bec-4653-af2f-bad8ffa75686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:16.539726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24702a3-0bec-4653-af2f-bad8ffa75686 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.540035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24702a3-0bec-4653-af2f-bad8ffa75686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.540308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24702a3-0bec-4653-af2f-bad8ffa75686 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.540948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24702a3-0bec-4653-af2f-bad8ffa75686 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:16.547906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24702a3-0bec-4653-af2f-bad8ffa75686 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:16.554896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1033/2054] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:43:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:43:16.556535 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:16.559494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24702a3-0bec-4653-af2f-bad8ffa75686 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.571001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24702a3-0bec-4653-af2f-bad8ffa75686 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:16.576932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338fbe0c-1b90-4f60-b80a-7855aa676075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:16.577606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338fbe0c-1b90-4f60-b80a-7855aa676075 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.577921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338fbe0c-1b90-4f60-b80a-7855aa676075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.578207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338fbe0c-1b90-4f60-b80a-7855aa676075 admin admin] PATH_INFO: `/v3/projects/e99220401d7e45348f96b9b0a69fa2d7/users/0f8d8556c0a04f1499d1ea234d5193c6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.589426 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc35a1a-5095-4800-a072-2e716165ef2a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:16.596827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7d806-a371-4e12-927f-453a853d2892 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:16.597553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7d806-a371-4e12-927f-453a853d2892 service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.597868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7d806-a371-4e12-927f-453a853d2892 service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.598219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7d806-a371-4e12-927f-453a853d2892 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.598620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338fbe0c-1b90-4f60-b80a-7855aa676075 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e99220401d7e45348f96b9b0a69fa2d7, user_id=0f8d8556c0a04f1499d1ea234d5193c6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:16.601432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338fbe0c-1b90-4f60-b80a-7855aa676075 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:16.608908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c7d806-a371-4e12-927f-453a853d2892 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:16.610847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c7d806-a371-4e12-927f-453a853d2892 service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:16.616960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1034/2055] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:43:16 2022] PUT /identity/v3/projects/e99220401d7e45348f96b9b0a69fa2d7/users/0f8d8556c0a04f1499d1ea234d5193c6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:43:16.619875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1022/2056] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:43:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:43:16.621416 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338fbe0c-1b90-4f60-b80a-7855aa676075 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.704818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338fbe0c-1b90-4f60-b80a-7855aa676075 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:16.710191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a840a7af-00f5-4026-bcca-c51b3fd49eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:16.710892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a840a7af-00f5-4026-bcca-c51b3fd49eff admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.711207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a840a7af-00f5-4026-bcca-c51b3fd49eff admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.711491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a840a7af-00f5-4026-bcca-c51b3fd49eff admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.712163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a840a7af-00f5-4026-bcca-c51b3fd49eff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:16.713916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a840a7af-00f5-4026-bcca-c51b3fd49eff admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:16.726670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1035/2057] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:43:16 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:43:16.730934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7d806-a371-4e12-927f-453a853d2892 service swift] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.743831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7d806-a371-4e12-927f-453a853d2892 service swift] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:16.750005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:16.750830 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.751537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] PATH_INFO: `/v3/projects/e99220401d7e45348f96b9b0a69fa2d7/users/0f8d8556c0a04f1499d1ea234d5193c6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.753669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] RBAC: Authorizing `identity:create_grant(project_id=e99220401d7e45348f96b9b0a69fa2d7, user_id=0f8d8556c0a04f1499d1ea234d5193c6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:16.757062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:16.771466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1023/2058] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:43:16 2022] PUT /identity/v3/projects/e99220401d7e45348f96b9b0a69fa2d7/users/0f8d8556c0a04f1499d1ea234d5193c6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:16.784469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87720d8-109b-42b0-b0e6-5351805419f7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:16.785113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87720d8-109b-42b0-b0e6-5351805419f7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:16.785418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87720d8-109b-42b0-b0e6-5351805419f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:16.861254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c87720d8-109b-42b0-b0e6-5351805419f7 None None] MFA Rules not processed for user `0f8d8556c0a04f1499d1ea234d5193c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:16.865459 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c87720d8-109b-42b0-b0e6-5351805419f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:16.917961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:16.975632 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1036/2059] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:16 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:16.991861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:16.994273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:17.103267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d142f29d-f58f-4137-8181-ec614619f3df admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:17.109783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92db032d-db58-4573-ad10-2dd5ebccf0d7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:17.110788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db032d-db58-4573-ad10-2dd5ebccf0d7 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:17.111363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db032d-db58-4573-ad10-2dd5ebccf0d7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:17.111728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db032d-db58-4573-ad10-2dd5ebccf0d7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:17.182970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92db032d-db58-4573-ad10-2dd5ebccf0d7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:17.184969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92db032d-db58-4573-ad10-2dd5ebccf0d7 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:17.202728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1024/2060] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:16 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:23.366556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87720d8-109b-42b0-b0e6-5351805419f7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:23.368052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c87720d8-109b-42b0-b0e6-5351805419f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:23.390096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87720d8-109b-42b0-b0e6-5351805419f7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:23.394293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019f6c59-f571-4862-baa5-00fed0698cd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:23.395062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f6c59-f571-4862-baa5-00fed0698cd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:23.395752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f6c59-f571-4862-baa5-00fed0698cd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:23.396045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f6c59-f571-4862-baa5-00fed0698cd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:23.411982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019f6c59-f571-4862-baa5-00fed0698cd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:23.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019f6c59-f571-4862-baa5-00fed0698cd3 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:23.430617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1037/2061] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:23 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:25.958651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b63463-5c94-4c28-b177-4e88716a8aa6 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:25.959291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b63463-5c94-4c28-b177-4e88716a8aa6 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:25.959291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b63463-5c94-4c28-b177-4e88716a8aa6 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:25.960710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1025/2062] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:43:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:43:25.969748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3cd581-7777-4a82-87fa-e83688bab44c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:25.971448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3cd581-7777-4a82-87fa-e83688bab44c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:25.971917 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3cd581-7777-4a82-87fa-e83688bab44c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:26.018871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-cf3cd581-7777-4a82-87fa-e83688bab44c None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:26.020536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cf3cd581-7777-4a82-87fa-e83688bab44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:26.157038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1038/2063] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:43:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:26.209383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b63463-5c94-4c28-b177-4e88716a8aa6 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:26.211420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-68b63463-5c94-4c28-b177-4e88716a8aa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:26.439773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b63463-5c94-4c28-b177-4e88716a8aa6 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:26.452650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40254fd-2dcb-4efb-8dc6-9170f76e3862 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e3f6be584e34bfb9e430c7756d071d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:26.453640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40254fd-2dcb-4efb-8dc6-9170f76e3862 service glance] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:26.454956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40254fd-2dcb-4efb-8dc6-9170f76e3862 service glance] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:26.455473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40254fd-2dcb-4efb-8dc6-9170f76e3862 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:26.469282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40254fd-2dcb-4efb-8dc6-9170f76e3862 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:26.471652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40254fd-2dcb-4efb-8dc6-9170f76e3862 service glance] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:26.485596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1026/2064] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:26.591225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351a5f1d-39a0-4fbe-90a1-e6349e373657 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:26.591484 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351a5f1d-39a0-4fbe-90a1-e6349e373657 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:26.591860 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351a5f1d-39a0-4fbe-90a1-e6349e373657 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:26.594300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1039/2065] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:43:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:43:26.601095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e22dae3-6afa-493e-bceb-848306e0021e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:26.601785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e22dae3-6afa-493e-bceb-848306e0021e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:26.602857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e22dae3-6afa-493e-bceb-848306e0021e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:26.618852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:26.635483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-3e22dae3-6afa-493e-bceb-848306e0021e None None] MFA Rules not processed for user `5756f4ff604a413b8af1f12c82cc4870`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:26.636812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3e22dae3-6afa-493e-bceb-848306e0021e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:26.705561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1027/2066] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:43:26 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:26.719014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351a5f1d-39a0-4fbe-90a1-e6349e373657 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:26.721231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-351a5f1d-39a0-4fbe-90a1-e6349e373657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:26.739557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351a5f1d-39a0-4fbe-90a1-e6349e373657 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:26.750097 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1254407f-7b21-47ff-baff-24f3aa6f269f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:26.750946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1254407f-7b21-47ff-baff-24f3aa6f269f service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:26.751371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1254407f-7b21-47ff-baff-24f3aa6f269f service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:26.751786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1254407f-7b21-47ff-baff-24f3aa6f269f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:26.764729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1254407f-7b21-47ff-baff-24f3aa6f269f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:26.767263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1254407f-7b21-47ff-baff-24f3aa6f269f service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:26.777792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1040/2067] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:26.900943 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387cc2a3-46fc-43b3-994b-6dea978511d7 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:26.901307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387cc2a3-46fc-43b3-994b-6dea978511d7 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:26.901564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387cc2a3-46fc-43b3-994b-6dea978511d7 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:26.903615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1028/2068] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:43:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:43:26.909770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ffe80-89d3-400a-83e0-fb993d3cdb74 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:26.910393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ffe80-89d3-400a-83e0-fb993d3cdb74 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:26.910884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0ffe80-89d3-400a-83e0-fb993d3cdb74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:26.941223 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-2b0ffe80-89d3-400a-83e0-fb993d3cdb74 None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:26.942743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2b0ffe80-89d3-400a-83e0-fb993d3cdb74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:26.957846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1041/2069] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:43:26 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:26.972100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387cc2a3-46fc-43b3-994b-6dea978511d7 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:26.973869 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-387cc2a3-46fc-43b3-994b-6dea978511d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:26.989817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387cc2a3-46fc-43b3-994b-6dea978511d7 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:26.994780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bb1c10-c5d5-42ed-ae75-706caf58690c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:26.995736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bb1c10-c5d5-42ed-ae75-706caf58690c service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:26.996067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bb1c10-c5d5-42ed-ae75-706caf58690c service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:26.996337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bb1c10-c5d5-42ed-ae75-706caf58690c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:27.023035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bb1c10-c5d5-42ed-ae75-706caf58690c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:27.024757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bb1c10-c5d5-42ed-ae75-706caf58690c service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:27.041390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1029/2070] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:35.849927 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc0f055-8526-4083-bc95-06a2d3891460 None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:35.850949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc0f055-8526-4083-bc95-06a2d3891460 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:35.850949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc0f055-8526-4083-bc95-06a2d3891460 None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:35.852514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1042/2071] 10.222.0.26 () {54 vars in 961 bytes} [Wed Oct 19 00:43:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:43:35.859677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5174a91e-2ee5-4d3d-90aa-81eeb46bd759 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:35.860618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5174a91e-2ee5-4d3d-90aa-81eeb46bd759 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:35.860618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5174a91e-2ee5-4d3d-90aa-81eeb46bd759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:35.882718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5174a91e-2ee5-4d3d-90aa-81eeb46bd759 None None] MFA Rules not processed for user `93d2a83a28fd4608994317c2c7b6a723`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:35.884028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5174a91e-2ee5-4d3d-90aa-81eeb46bd759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:35.961206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1030/2072] 10.222.0.26 () {58 vars in 1058 bytes} [Wed Oct 19 00:43:35 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:35.969745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc0f055-8526-4083-bc95-06a2d3891460 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:35.984726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc0f055-8526-4083-bc95-06a2d3891460 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:35.991924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:35.992975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:35.993456 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:35.993852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:35.996877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:36.009661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:36.011309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:36.023232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1043/2073] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:36.287085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88606e2e-e87e-48b6-ba56-74e68448b9cc None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:36.287353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88606e2e-e87e-48b6-ba56-74e68448b9cc None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:36.287655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88606e2e-e87e-48b6-ba56-74e68448b9cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:36.323334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-88606e2e-e87e-48b6-ba56-74e68448b9cc None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:36.328748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-88606e2e-e87e-48b6-ba56-74e68448b9cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:36.391826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1031/2074] 10.222.0.26 () {58 vars in 967 bytes} [Wed Oct 19 00:43:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:36.406039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:36.491948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b00eeb6-116d-49d0-918b-2dff3c0e326c service placement] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:36.498910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e25701-8c4f-4140-a7b0-47437e8de15f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:36.499615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e25701-8c4f-4140-a7b0-47437e8de15f service swift] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:36.500010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e25701-8c4f-4140-a7b0-47437e8de15f service swift] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:36.500351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e25701-8c4f-4140-a7b0-47437e8de15f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:36.514612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e25701-8c4f-4140-a7b0-47437e8de15f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:36.516227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e25701-8c4f-4140-a7b0-47437e8de15f service swift] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:36.520150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:36.526607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1044/2075] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:43:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:43:39.660363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dbd6fe-679e-4366-a727-5fce0368d259 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:39.661242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dbd6fe-679e-4366-a727-5fce0368d259 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:39.661377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dbd6fe-679e-4366-a727-5fce0368d259 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:39.703538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d3dbd6fe-679e-4366-a727-5fce0368d259 None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:39.704904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d3dbd6fe-679e-4366-a727-5fce0368d259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:39.713596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1032/2076] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:43:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:39.728960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e25701-8c4f-4140-a7b0-47437e8de15f service swift] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:39.749782 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e25701-8c4f-4140-a7b0-47437e8de15f service swift] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:39.758357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb8f57-a92a-4c2c-bf31-f91646f94b53 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:39.759928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb8f57-a92a-4c2c-bf31-f91646f94b53 service swift] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:39.759928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb8f57-a92a-4c2c-bf31-f91646f94b53 service swift] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:39.759928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb8f57-a92a-4c2c-bf31-f91646f94b53 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:39.803293 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fb8f57-a92a-4c2c-bf31-f91646f94b53 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:39.805003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fb8f57-a92a-4c2c-bf31-f91646f94b53 service swift] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:39.823521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1045/2077] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:43:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 00:43:42.015079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dbd6fe-679e-4366-a727-5fce0368d259 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:42.017202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d3dbd6fe-679e-4366-a727-5fce0368d259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:42.129442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:42.242874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dbd6fe-679e-4366-a727-5fce0368d259 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:42.264410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:42.265297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:42.265756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:42.266202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] PATH_INFO: `/v3/users/a08c9457231a447eb1c4130e68cfb31d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:42.267744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a08c9457231a447eb1c4130e68cfb31d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:42.269969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:42.355971 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a08c9457231a447eb1c4130e68cfb31d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:42.371336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a08c9457231a447eb1c4130e68cfb31d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:42.372816 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a08c9457231a447eb1c4130e68cfb31d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:42.377434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a08c9457231a447eb1c4130e68cfb31d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:42.394050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a08c9457231a447eb1c4130e68cfb31d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:42.447258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1033/2078] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:43:42 2022] DELETE /identity/v3/users/a08c9457231a447eb1c4130e68cfb31d => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:42.895222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb8f57-a92a-4c2c-bf31-f91646f94b53 service swift] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:42.961045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb8f57-a92a-4c2c-bf31-f91646f94b53 service swift] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:43.215218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:43.216121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:43.216787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:43.217269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] PATH_INFO: `/v3/projects/173a52b79dab4962a4aa380c26f97d2d` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:43.218897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=173a52b79dab4962a4aa380c26f97d2d)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:43.233889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:43.383199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '173a52b79dab4962a4aa380c26f97d2d', 'request_id': 'req-bc5b82d6-76e8-4069-bdb1-d309902d0e58'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:43.423980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '173a52b79dab4962a4aa380c26f97d2d', 'request_id': 'req-bc5b82d6-76e8-4069-bdb1-d309902d0e58'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:43.453052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '173a52b79dab4962a4aa380c26f97d2d', 'request_id': 'req-bc5b82d6-76e8-4069-bdb1-d309902d0e58'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:43.518526 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] The token cache is being invalidate because project 173a52b79dab4962a4aa380c26f97d2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:43:43.519104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:43.520586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:43.525567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1046/2079] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:43:42 2022] DELETE /identity/v3/projects/173a52b79dab4962a4aa380c26f97d2d => generated 0 bytes in 643 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:51.816788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:51.826473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:51.843367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae3796-d215-49aa-bd15-a4ec804d3758 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:51.925728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:51.926291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:51.926603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:51.926878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] PATH_INFO: `/v3/users/ed3d6b4184274791bce9c7d9b1ab028a` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:51.928105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed3d6b4184274791bce9c7d9b1ab028a)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:51.930055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:51.987176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed3d6b4184274791bce9c7d9b1ab028a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:51.997521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed3d6b4184274791bce9c7d9b1ab028a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:51.998333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed3d6b4184274791bce9c7d9b1ab028a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:51.999053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed3d6b4184274791bce9c7d9b1ab028a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.001345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:52.009617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed3d6b4184274791bce9c7d9b1ab028a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.033256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1034/2080] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:43:51 2022] DELETE /identity/v3/users/ed3d6b4184274791bce9c7d9b1ab028a => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:52.258000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:52.260369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:52.283842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5b82d6-76e8-4069-bdb1-d309902d0e58 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:52.349224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:52.351615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:52.394389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:52.395485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:52.396085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:52.396529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] PATH_INFO: `/v3/projects/f798b16ae6e4429c9c07793a0178f798` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:52.398554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f798b16ae6e4429c9c07793a0178f798)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:52.403472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:52.437574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:52.472831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54435bb4-6828-49ff-800d-79ebc958d529 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:52.478419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:52.484820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:52.485139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:52.485346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] PATH_INFO: `/v3/users/1dcc438968d64085a15be58c5a049718` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:52.487170 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] RBAC: Authorizing `identity:delete_user(user_id=1dcc438968d64085a15be58c5a049718)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:52.488801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:52.499125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f798b16ae6e4429c9c07793a0178f798', 'request_id': 'req-dd4b0135-5c99-477f-b7dc-726fcfb6e621'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f798b16ae6e4429c9c07793a0178f798', 'request_id': 'req-dd4b0135-5c99-477f-b7dc-726fcfb6e621'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.526353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f798b16ae6e4429c9c07793a0178f798', 'request_id': 'req-dd4b0135-5c99-477f-b7dc-726fcfb6e621'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.550835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] The token cache is being invalidate because project f798b16ae6e4429c9c07793a0178f798 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:43:52.551312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.552160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.554315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1047/2081] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:43:52 2022] DELETE /identity/v3/projects/f798b16ae6e4429c9c07793a0178f798 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:43:52.559843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:52.561615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:52.576333 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1dcc438968d64085a15be58c5a049718'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.591702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1dcc438968d64085a15be58c5a049718'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.592865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1dcc438968d64085a15be58c5a049718'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.593844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1dcc438968d64085a15be58c5a049718'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.606742 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e072c2cb-c0be-4572-a559-d97e1874310c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1dcc438968d64085a15be58c5a049718'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.638014 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1035/2082] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:43:52 2022] DELETE /identity/v3/users/1dcc438968d64085a15be58c5a049718 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:52.644487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ccc355-2768-4324-abc0-19cb88e04dde None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:52.644907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ccc355-2768-4324-abc0-19cb88e04dde None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:52.645275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ccc355-2768-4324-abc0-19cb88e04dde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:52.657692 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b0135-5c99-477f-b7dc-726fcfb6e621 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:52.669693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a9ccc355-2768-4324-abc0-19cb88e04dde None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:52.671292 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a9ccc355-2768-4324-abc0-19cb88e04dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:52.737620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:52.744747 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:52.745462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:52.746382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:52.746740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] PATH_INFO: `/v3/users/fb124845696e4ed6ad0b7c566f8bbd50` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:52.748113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb124845696e4ed6ad0b7c566f8bbd50)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:52.750270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:52.758732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1036/2083] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:43:52.774238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ccc355-2768-4324-abc0-19cb88e04dde None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:52.776343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a9ccc355-2768-4324-abc0-19cb88e04dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:52.830857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb124845696e4ed6ad0b7c566f8bbd50'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.839123 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:52.845296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb124845696e4ed6ad0b7c566f8bbd50'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.846618 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb124845696e4ed6ad0b7c566f8bbd50'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.847690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb124845696e4ed6ad0b7c566f8bbd50'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.862382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ccc355-2768-4324-abc0-19cb88e04dde None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:52.863294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb124845696e4ed6ad0b7c566f8bbd50'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:52.867969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:52.868747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:52.869104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:52.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:52.873174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:52.888343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:52.889851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1048/2084] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:43:52 2022] DELETE /identity/v3/users/fb124845696e4ed6ad0b7c566f8bbd50 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:43:52.893198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:52.907311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1037/2085] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:43:53.178248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:53.181164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:53.269643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946f7f71-e084-4e67-a8da-97b4fbc403e1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:53.274818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:53.275477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:53.275757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:53.276034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] PATH_INFO: `/v3/projects/995ba646a2c047c5a9b1df24325ff98d` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:53.277343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] RBAC: Authorizing `identity:delete_project(project_id=995ba646a2c047c5a9b1df24325ff98d)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:53.279348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:53.347425 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '995ba646a2c047c5a9b1df24325ff98d', 'request_id': 'req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.349906 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:53.352373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:53.356331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '995ba646a2c047c5a9b1df24325ff98d', 'request_id': 'req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.367316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '995ba646a2c047c5a9b1df24325ff98d', 'request_id': 'req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.376974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304bee4-9489-42d0-88ca-27529654122b service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:53.383185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] The token cache is being invalidate because project 995ba646a2c047c5a9b1df24325ff98d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:43:53.383813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.384601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.386301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1049/2086] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:43:53 2022] DELETE /identity/v3/projects/995ba646a2c047c5a9b1df24325ff98d => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:53.488468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:53.489233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:53.489691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:53.490174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] PATH_INFO: `/v3/projects/414577d72f964d759d66d26cf8f55a96` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:53.491915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] RBAC: Authorizing `identity:delete_project(project_id=414577d72f964d759d66d26cf8f55a96)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:53.494346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:53.578359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '414577d72f964d759d66d26cf8f55a96', 'request_id': 'req-ed6783be-4cda-4724-9d2e-6601f852096b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.596434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '414577d72f964d759d66d26cf8f55a96', 'request_id': 'req-ed6783be-4cda-4724-9d2e-6601f852096b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.610351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '414577d72f964d759d66d26cf8f55a96', 'request_id': 'req-ed6783be-4cda-4724-9d2e-6601f852096b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.638613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] The token cache is being invalidate because project 414577d72f964d759d66d26cf8f55a96 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:43:53.639118 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.640177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.641799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1038/2087] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:43:53 2022] DELETE /identity/v3/projects/414577d72f964d759d66d26cf8f55a96 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:53.647077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:53.649644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:53.675013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3f7783-3f0f-4cab-a70f-9bd88ba28687 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:53.767514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:53.768286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:53.768663 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:53.769121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] PATH_INFO: `/v3/users/0f8d8556c0a04f1499d1ea234d5193c6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:53.770724 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f8d8556c0a04f1499d1ea234d5193c6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:53.773826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:53.778289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:53.849795 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f8d8556c0a04f1499d1ea234d5193c6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.860010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f8d8556c0a04f1499d1ea234d5193c6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.860914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f8d8556c0a04f1499d1ea234d5193c6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.861851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f8d8556c0a04f1499d1ea234d5193c6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.872057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-ffa4cfbb-e165-4d04-ae39-ac53e568c61e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f8d8556c0a04f1499d1ea234d5193c6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:53.895453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1050/2088] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:43:53 2022] DELETE /identity/v3/users/0f8d8556c0a04f1499d1ea234d5193c6 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:54.289146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:54.291281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:54.311206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6783be-4cda-4724-9d2e-6601f852096b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:54.351352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:54.398193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:54.398796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:54.399081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:54.399386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] PATH_INFO: `/v3/projects/e99220401d7e45348f96b9b0a69fa2d7` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:54.400746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e99220401d7e45348f96b9b0a69fa2d7)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:54.402838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:54.456792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e99220401d7e45348f96b9b0a69fa2d7', 'request_id': 'req-55cc3441-d36e-4566-a29c-ae189dd89da7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:54.464206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e99220401d7e45348f96b9b0a69fa2d7', 'request_id': 'req-55cc3441-d36e-4566-a29c-ae189dd89da7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:54.471509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e99220401d7e45348f96b9b0a69fa2d7', 'request_id': 'req-55cc3441-d36e-4566-a29c-ae189dd89da7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:54.488458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] The token cache is being invalidate because project e99220401d7e45348f96b9b0a69fa2d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:43:54.488892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:54.489728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-55cc3441-d36e-4566-a29c-ae189dd89da7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:43:54.491444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1039/2089] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:43:54 2022] DELETE /identity/v3/projects/e99220401d7e45348f96b9b0a69fa2d7 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:54.496988 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00a0f7-d191-4219-987c-cef8c8ecc717 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:54.497628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00a0f7-d191-4219-987c-cef8c8ecc717 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:54.497628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00a0f7-d191-4219-987c-cef8c8ecc717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:54.521055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-3a00a0f7-d191-4219-987c-cef8c8ecc717 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:54.522467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3a00a0f7-d191-4219-987c-cef8c8ecc717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:54.601748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1051/2090] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:43:54.611137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78d2da-6586-4ef3-8f99-861d74bf5514 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:54.611590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78d2da-6586-4ef3-8f99-861d74bf5514 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:54.612002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78d2da-6586-4ef3-8f99-861d74bf5514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:54.635049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8a78d2da-6586-4ef3-8f99-861d74bf5514 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:54.636814 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8a78d2da-6586-4ef3-8f99-861d74bf5514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:54.645453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1040/2091] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:43:54.650710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a00a0f7-d191-4219-987c-cef8c8ecc717 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:54.662044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a00a0f7-d191-4219-987c-cef8c8ecc717 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:54.667100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:54.667986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:54.668643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:54.668643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:54.669075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:43:54.669379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:54.671118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:54.685323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1052/2092] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:43:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:43:54.690017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78d2da-6586-4ef3-8f99-861d74bf5514 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:54.700501 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78d2da-6586-4ef3-8f99-861d74bf5514 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:54.705610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01049d8-d6b9-4d3b-8c57-972d1993b499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:54.706222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01049d8-d6b9-4d3b-8c57-972d1993b499 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:54.706530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01049d8-d6b9-4d3b-8c57-972d1993b499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:54.706806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01049d8-d6b9-4d3b-8c57-972d1993b499 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:54.707489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01049d8-d6b9-4d3b-8c57-972d1993b499 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:54.709480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01049d8-d6b9-4d3b-8c57-972d1993b499 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:54.741461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1041/2093] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:43:54 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:43:54.746938 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:54.757557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57686de-f6cd-4466-bb54-647bc12ae02c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:54.825555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ef6280-1e22-4657-bfa4-b730fe149e19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:54.826386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef6280-1e22-4657-bfa4-b730fe149e19 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:54.826899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef6280-1e22-4657-bfa4-b730fe149e19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:54.827531 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef6280-1e22-4657-bfa4-b730fe149e19 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:54.828282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ef6280-1e22-4657-bfa4-b730fe149e19 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:54.831264 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ef6280-1e22-4657-bfa4-b730fe149e19 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:54.855347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:54.868807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1053/2094] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:43:54 2022] POST /identity/v3/users => generated 290 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:43:54.876184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01049d8-d6b9-4d3b-8c57-972d1993b499 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:54.904467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01049d8-d6b9-4d3b-8c57-972d1993b499 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:54.910910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67319bc-5518-4215-921a-a16b0917c171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:54.911546 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67319bc-5518-4215-921a-a16b0917c171 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:54.911821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67319bc-5518-4215-921a-a16b0917c171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:54.912097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67319bc-5518-4215-921a-a16b0917c171 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:54.912763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67319bc-5518-4215-921a-a16b0917c171 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:54.914544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67319bc-5518-4215-921a-a16b0917c171 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:54.931427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1042/2095] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:43:54 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:43:54.936442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ef6280-1e22-4657-bfa4-b730fe149e19 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:54.955388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ef6280-1e22-4657-bfa4-b730fe149e19 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:54.963553 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267951ea-4baf-4ab7-9aad-d67f013de001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:54.964679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267951ea-4baf-4ab7-9aad-d67f013de001 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:54.965363 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267951ea-4baf-4ab7-9aad-d67f013de001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:54.966045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267951ea-4baf-4ab7-9aad-d67f013de001 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:54.967288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267951ea-4baf-4ab7-9aad-d67f013de001 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:54.970738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267951ea-4baf-4ab7-9aad-d67f013de001 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:54.995037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1054/2096] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:43:54 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:43:54.997223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67319bc-5518-4215-921a-a16b0917c171 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:55.007987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67319bc-5518-4215-921a-a16b0917c171 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:55.015018 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:55.015917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:55.016200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:55.016479 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] PATH_INFO: `/v3/projects/8684504b0eff415b9063cebae21ca667/users/f738e67d8c2340028da1254b60d7810e/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:55.036533 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=8684504b0eff415b9063cebae21ca667, user_id=f738e67d8c2340028da1254b60d7810e, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:55.041865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:55.064647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1043/2097] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:43:54 2022] PUT /identity/v3/projects/8684504b0eff415b9063cebae21ca667/users/f738e67d8c2340028da1254b60d7810e/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:43:55.070586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa33f49-9fac-4d71-9a71-69cc2dc35140 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:55.070943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa33f49-9fac-4d71-9a71-69cc2dc35140 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:55.071265 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa33f49-9fac-4d71-9a71-69cc2dc35140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:55.095407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5aa33f49-9fac-4d71-9a71-69cc2dc35140 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:43:55.096666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5aa33f49-9fac-4d71-9a71-69cc2dc35140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:55.185953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1055/2098] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:43:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:43:55.196813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:43:55.199265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:43:55.283466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca43ff-e78d-4b35-bf43-5095c1cbaf3c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:43:55.287859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c93b32-eae2-401f-95f1-223830bf83ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:43:55.288498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c93b32-eae2-401f-95f1-223830bf83ea service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:43:55.288775 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c93b32-eae2-401f-95f1-223830bf83ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:43:55.289041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c93b32-eae2-401f-95f1-223830bf83ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:43:55.297906 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:43:55.301765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c93b32-eae2-401f-95f1-223830bf83ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:43:55.304650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c93b32-eae2-401f-95f1-223830bf83ea service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:43:55.319276 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1044/2099] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:43:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:44:01.104031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5c8403-db0f-4cc9-a65f-7fa8a3dd2bcf None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:01.105323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5c8403-db0f-4cc9-a65f-7fa8a3dd2bcf None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:01.105323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5c8403-db0f-4cc9-a65f-7fa8a3dd2bcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:01.147976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-6e5c8403-db0f-4cc9-a65f-7fa8a3dd2bcf None None] MFA Rules not processed for user `f738e67d8c2340028da1254b60d7810e`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:44:01.149240 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-6e5c8403-db0f-4cc9-a65f-7fa8a3dd2bcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:01.270732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1056/2100] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:44:01 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:44:01.275513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c93b32-eae2-401f-95f1-223830bf83ea service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:44:01.287758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c93b32-eae2-401f-95f1-223830bf83ea service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:44:01.293596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd18967-208d-43a4-b389-2ef3edc7ea6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:44:01.294224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd18967-208d-43a4-b389-2ef3edc7ea6e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:01.294523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd18967-208d-43a4-b389-2ef3edc7ea6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:01.294797 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd18967-208d-43a4-b389-2ef3edc7ea6e admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:01.295645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd18967-208d-43a4-b389-2ef3edc7ea6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:44:01.297682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd18967-208d-43a4-b389-2ef3edc7ea6e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:44:01.350412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1045/2101] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:44:01 2022] POST /identity/v3/projects => generated 379 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:44:01.351508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5c8403-db0f-4cc9-a65f-7fa8a3dd2bcf None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:44:01.373201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5c8403-db0f-4cc9-a65f-7fa8a3dd2bcf None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:44:01.528249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49df2c02-b82c-4408-a6d9-fc3f1e99fcb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:44:01.529378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df2c02-b82c-4408-a6d9-fc3f1e99fcb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:01.529749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df2c02-b82c-4408-a6d9-fc3f1e99fcb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:01.530221 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df2c02-b82c-4408-a6d9-fc3f1e99fcb0 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:01.531466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49df2c02-b82c-4408-a6d9-fc3f1e99fcb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:44:01.535090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49df2c02-b82c-4408-a6d9-fc3f1e99fcb0 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:44:01.582668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1057/2102] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:44:01 2022] POST /identity/v3/users => generated 296 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:44:01.588747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd18967-208d-43a4-b389-2ef3edc7ea6e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:44:01.624409 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd18967-208d-43a4-b389-2ef3edc7ea6e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:44:01.634583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192edc26-aeca-402c-968b-cb811c02b000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:44:01.636124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192edc26-aeca-402c-968b-cb811c02b000 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:01.636627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192edc26-aeca-402c-968b-cb811c02b000 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:01.637133 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192edc26-aeca-402c-968b-cb811c02b000 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:01.638784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192edc26-aeca-402c-968b-cb811c02b000 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:44:01.644591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192edc26-aeca-402c-968b-cb811c02b000 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:44:01.659736 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1046/2103] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:44:01 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:44:01.666382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49df2c02-b82c-4408-a6d9-fc3f1e99fcb0 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:44:01.688344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49df2c02-b82c-4408-a6d9-fc3f1e99fcb0 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:44:01.695816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74f4b33-1c1a-406d-81d5-22cc70333b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:44:01.696671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74f4b33-1c1a-406d-81d5-22cc70333b65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:01.696960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74f4b33-1c1a-406d-81d5-22cc70333b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:01.697236 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74f4b33-1c1a-406d-81d5-22cc70333b65 admin admin] PATH_INFO: `/v3/projects/a00f442306cf489f9f7c28839bec1d0e/users/e9307ff0301248448ef8b83befda1cf6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:01.700790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:44:01.739039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74f4b33-1c1a-406d-81d5-22cc70333b65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a00f442306cf489f9f7c28839bec1d0e, user_id=e9307ff0301248448ef8b83befda1cf6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:44:01.744783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74f4b33-1c1a-406d-81d5-22cc70333b65 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:44:01.770985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1058/2104] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:44:01 2022] PUT /identity/v3/projects/a00f442306cf489f9f7c28839bec1d0e/users/e9307ff0301248448ef8b83befda1cf6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:44:01.774101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192edc26-aeca-402c-968b-cb811c02b000 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:44:01.836106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192edc26-aeca-402c-968b-cb811c02b000 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:44:01.985228 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d2c53-651c-47f0-9cc4-b3336bcf1560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:44:01.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d2c53-651c-47f0-9cc4-b3336bcf1560 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:01.987613 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d2c53-651c-47f0-9cc4-b3336bcf1560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:01.988101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d2c53-651c-47f0-9cc4-b3336bcf1560 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:01.989347 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06d2c53-651c-47f0-9cc4-b3336bcf1560 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:44:01.993265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06d2c53-651c-47f0-9cc4-b3336bcf1560 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:44:02.017589 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1047/2105] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:44:01 2022] GET /identity/v3/roles => generated 1420 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:44:02.019903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74f4b33-1c1a-406d-81d5-22cc70333b65 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:44:02.035474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74f4b33-1c1a-406d-81d5-22cc70333b65 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:44:02.049114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:44:02.050410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:02.050972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:02.051464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] PATH_INFO: `/v3/projects/a00f442306cf489f9f7c28839bec1d0e/users/e9307ff0301248448ef8b83befda1cf6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:02.055256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a00f442306cf489f9f7c28839bec1d0e, user_id=e9307ff0301248448ef8b83befda1cf6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:44:02.064564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:44:02.084057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1059/2106] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:44:02 2022] PUT /identity/v3/projects/a00f442306cf489f9f7c28839bec1d0e/users/e9307ff0301248448ef8b83befda1cf6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:44:09.448897 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3d6dbc-bc2d-4c87-9c60-ef11860af6a0 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:09.450452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3d6dbc-bc2d-4c87-9c60-ef11860af6a0 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:09.451033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3d6dbc-bc2d-4c87-9c60-ef11860af6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:09.491608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-9f3d6dbc-bc2d-4c87-9c60-ef11860af6a0 None None] MFA Rules not processed for user `e9307ff0301248448ef8b83befda1cf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:44:09.492894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9f3d6dbc-bc2d-4c87-9c60-ef11860af6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:09.649255 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1048/2107] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:44:09 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:44:09.671631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:44:09.673598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:09.806738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046cca40-675a-4241-b6e8-4c8f7f27260d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:44:09.816464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:44:09.817494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:09.817889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:09.818343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:09.942946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:44:09.947743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:44:09.964824 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1060/2108] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:44:09 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:44:13.463117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e026c5-ec92-4caa-9ca4-368e24d6076f None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:13.464298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e026c5-ec92-4caa-9ca4-368e24d6076f None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:13.464298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e026c5-ec92-4caa-9ca4-368e24d6076f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:13.533969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a1e026c5-ec92-4caa-9ca4-368e24d6076f None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:44:13.536621 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a1e026c5-ec92-4caa-9ca4-368e24d6076f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:13.692675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:44:13.726448 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1049/2109] 10.222.0.26 () {58 vars in 967 bytes} [Wed Oct 19 00:44:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:44:13.790390 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:44:13.794604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:14.064894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df618a9-69d9-4ce3-87ed-3ae81e7b62df service nova] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:44:14.088667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0efbca-62ca-4f9f-9633-80115d9d4f67 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:44:14.089343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0efbca-62ca-4f9f-9633-80115d9d4f67 service swift] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:14.089630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0efbca-62ca-4f9f-9633-80115d9d4f67 service swift] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:14.089907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0efbca-62ca-4f9f-9633-80115d9d4f67 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:14.110564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:44:14.116899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0efbca-62ca-4f9f-9633-80115d9d4f67 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:44:14.127571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0efbca-62ca-4f9f-9633-80115d9d4f67 service swift] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:44:14.151331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1061/2110] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:44:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:44:23.711421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:44:24.139907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:44:31.645419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057df14-2438-45ad-9475-d5ed86e41c04 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:31.646039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057df14-2438-45ad-9475-d5ed86e41c04 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:31.646561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0057df14-2438-45ad-9475-d5ed86e41c04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:31.680200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-0057df14-2438-45ad-9475-d5ed86e41c04 None None] MFA Rules not processed for user `f738e67d8c2340028da1254b60d7810e`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:44:31.681832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0057df14-2438-45ad-9475-d5ed86e41c04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:31.694048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1050/2111] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:44:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:44:32.779023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e38b0e1-0a26-49d0-82d8-73c7ae5cf411 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:32.779925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e38b0e1-0a26-49d0-82d8-73c7ae5cf411 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:32.779925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e38b0e1-0a26-49d0-82d8-73c7ae5cf411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:32.807164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-7e38b0e1-0a26-49d0-82d8-73c7ae5cf411 None None] MFA Rules not processed for user `6f8725c4d05b4dd79b05b43fba15980b`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:44:32.808637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7e38b0e1-0a26-49d0-82d8-73c7ae5cf411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:32.939095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1062/2112] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:44:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:44:41.672626 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:44:42.925802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=25 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:44:44.749822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add371f9-ac32-4c6a-b3f3-4f1f4293f34b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:44.750439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add371f9-ac32-4c6a-b3f3-4f1f4293f34b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:44.750825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add371f9-ac32-4c6a-b3f3-4f1f4293f34b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:44.778336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-add371f9-ac32-4c6a-b3f3-4f1f4293f34b None None] MFA Rules not processed for user `39de6a6fff7c4cc08cc62aa86182d71d`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:44:44.780150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-add371f9-ac32-4c6a-b3f3-4f1f4293f34b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:44.875619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1051/2113] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:44:44 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:44:50.916467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be12ce-4abf-47c7-aa48-dd49cf5756e4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:44:50.917026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be12ce-4abf-47c7-aa48-dd49cf5756e4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:44:50.917509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be12ce-4abf-47c7-aa48-dd49cf5756e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:44:50.940104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-d0be12ce-4abf-47c7-aa48-dd49cf5756e4 None None] MFA Rules not processed for user `d3f0d615187d4779bdd7da8c9f24c796`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:44:50.941460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d0be12ce-4abf-47c7-aa48-dd49cf5756e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:44:51.023297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1063/2114] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:44:50 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:44:54.867162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=25 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:45:00.256889 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add371f9-ac32-4c6a-b3f3-4f1f4293f34b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:45:00.357696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add371f9-ac32-4c6a-b3f3-4f1f4293f34b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:45:00.364009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:45:00.365053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:45:00.365543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:45:00.366019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:45:00.384231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:45:00.387134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:45:00.406300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1052/2115] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:45:00 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:45:01.015864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=25 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:45:10.397723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=26 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:45:11.911959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9551c022-d498-4866-a02d-b4eb353f16d7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:45:11.912849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9551c022-d498-4866-a02d-b4eb353f16d7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:45:11.912849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9551c022-d498-4866-a02d-b4eb353f16d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:45:11.934412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9551c022-d498-4866-a02d-b4eb353f16d7 None None] MFA Rules not processed for user `69fef5650fff42e4ac4ed3ca74d4b123`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:45:11.935689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9551c022-d498-4866-a02d-b4eb353f16d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:45:11.997172 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1064/2116] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:45:11 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:45:21.989424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=25 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:45:24.667708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:45:24.669702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:45:24.684894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da4719c-98dd-4a37-bff4-41806909d33b service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:45:24.690175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:45:24.691386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:45:24.691894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:45:24.692387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:45:24.696598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:45:24.722828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:45:24.725682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:45:24.748220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1053/2117] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:45:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:45:34.737798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:45:40.036194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9551c022-d498-4866-a02d-b4eb353f16d7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:45:40.037637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9551c022-d498-4866-a02d-b4eb353f16d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:45:40.121638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9551c022-d498-4866-a02d-b4eb353f16d7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:45:40.125454 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:45:40.125980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:45:40.126242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:45:40.126532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:45:40.128972 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:45:40.143461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:45:40.144944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:45:40.156477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1065/2118] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:45:40 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:45:42.722088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:45:42.734950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7154c1-76e4-40cb-a394-ccc6cc5eb1ca service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:45:42.740664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:45:42.741285 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:45:42.741574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:45:42.741844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:45:42.744288 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:45:42.755679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:45:42.757283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:45:42.768107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1054/2119] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:45:42 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:45:50.151638 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=26 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:45:52.763329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:00.942998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:00.945639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:01.039555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:01.040609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:01.113855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da33eb3-3ab0-4670-bbb0-66fa3a672f85 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:01.128015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:01.129729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:01.130907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:01.132879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:01.136170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:01.157138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:01.160007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:01.175089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1066/2120] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:46:01.183563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c086fa14-b404-4d64-9ddd-ecc9a784879d service nova] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:01.192197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:01.193925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:01.195069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:01.196240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] PATH_INFO: `/v3/users/39de6a6fff7c4cc08cc62aa86182d71d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:01.198970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39de6a6fff7c4cc08cc62aa86182d71d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:01.202630 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:01.300014 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39de6a6fff7c4cc08cc62aa86182d71d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.311063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39de6a6fff7c4cc08cc62aa86182d71d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.311852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39de6a6fff7c4cc08cc62aa86182d71d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.312944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39de6a6fff7c4cc08cc62aa86182d71d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.325965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39de6a6fff7c4cc08cc62aa86182d71d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.368439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1055/2121] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:46:01 2022] DELETE /identity/v3/users/39de6a6fff7c4cc08cc62aa86182d71d => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:46:01.684475 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:01.705076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e67cc1-2ec1-4b99-abc2-86ad9fbcfd57 service placement] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:01.809225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:01.810036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:01.810433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:01.810843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] PATH_INFO: `/v3/projects/1235d8b054b3405595fca761f08b73ae` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:01.812235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] RBAC: Authorizing `identity:delete_project(project_id=1235d8b054b3405595fca761f08b73ae)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:01.814334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:01.871773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1235d8b054b3405595fca761f08b73ae', 'request_id': 'req-b70ede48-82c5-4765-bc08-ae2570ba2fea'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.879358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1235d8b054b3405595fca761f08b73ae', 'request_id': 'req-b70ede48-82c5-4765-bc08-ae2570ba2fea'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.889125 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1235d8b054b3405595fca761f08b73ae', 'request_id': 'req-b70ede48-82c5-4765-bc08-ae2570ba2fea'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] The token cache is being invalidate because project 1235d8b054b3405595fca761f08b73ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:46:01.904484 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.905497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:01.907201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1067/2122] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:46:01 2022] DELETE /identity/v3/projects/1235d8b054b3405595fca761f08b73ae => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:46:07.505640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:07.510169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:07.703181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec884315-4707-462e-8270-4f569f28a380 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:07.737612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:07.840251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08c7d61-7b02-4eac-9b0c-ffd23974ab16 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:07.841125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08c7d61-7b02-4eac-9b0c-ffd23974ab16 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:07.841761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08c7d61-7b02-4eac-9b0c-ffd23974ab16 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:07.842301 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08c7d61-7b02-4eac-9b0c-ffd23974ab16 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:07.844187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f08c7d61-7b02-4eac-9b0c-ffd23974ab16 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:07.868946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08c7d61-7b02-4eac-9b0c-ffd23974ab16 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:07.871158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08c7d61-7b02-4eac-9b0c-ffd23974ab16 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:07.892322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1056/2123] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:09.246780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:09.249808 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:09.292762 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:09.346718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70ede48-82c5-4765-bc08-ae2570ba2fea admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:09.353161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:09.354119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:09.354604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:09.355071 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:09.357909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:09.461042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:09.463329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:09.477842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1068/2124] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:46:13.278000 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b9f79-5105-45c5-bd95-70064f96a184 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:13.278728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b9f79-5105-45c5-bd95-70064f96a184 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:13.278728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b9f79-5105-45c5-bd95-70064f96a184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:13.303005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-4d0b9f79-5105-45c5-bd95-70064f96a184 None None] MFA Rules not processed for user `6f8725c4d05b4dd79b05b43fba15980b`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:13.305164 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4d0b9f79-5105-45c5-bd95-70064f96a184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:13.401325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1057/2125] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:46:13 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:46:14.118445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:14.142167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d86f82-732b-440f-808b-b5f273cd77f2 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:14.149340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:14.150709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:14.151321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:14.151907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:14.155324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:14.199340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:14.201058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:14.215261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1069/2126] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:14.615673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0b9f79-5105-45c5-bd95-70064f96a184 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:14.618820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4d0b9f79-5105-45c5-bd95-70064f96a184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:14.640537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0b9f79-5105-45c5-bd95-70064f96a184 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:14.645569 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:14.646239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:14.646537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:14.646824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] PATH_INFO: `/v3/users/d3f0d615187d4779bdd7da8c9f24c796` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:14.648838 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d3f0d615187d4779bdd7da8c9f24c796)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:14.651110 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:14.727566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd3f0d615187d4779bdd7da8c9f24c796'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:14.741315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd3f0d615187d4779bdd7da8c9f24c796'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:14.742593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd3f0d615187d4779bdd7da8c9f24c796'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:14.743597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd3f0d615187d4779bdd7da8c9f24c796'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:14.764794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd3f0d615187d4779bdd7da8c9f24c796'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:14.792238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:14.794049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:14.805125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1058/2127] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:46:14 2022] DELETE /identity/v3/users/d3f0d615187d4779bdd7da8c9f24c796 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:46:14.922562 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98811917-b9ac-4ef2-bc67-b7c6c2a97063 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:14.970697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:14.973250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:15.022170 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:15.042449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:15.079797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:15.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:15.081786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:15.082170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:15.085017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:15.100415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:15.102435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:15.115366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1070/2128] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:15.147769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a347e1-b3f0-4eec-ba80-8c860b4b1e88 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:15.159871 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8beca6ba-32d9-4136-af4f-cab3c5fcfe31 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:15.160620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beca6ba-32d9-4136-af4f-cab3c5fcfe31 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:15.160961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beca6ba-32d9-4136-af4f-cab3c5fcfe31 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:15.161313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8beca6ba-32d9-4136-af4f-cab3c5fcfe31 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:15.165407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8beca6ba-32d9-4136-af4f-cab3c5fcfe31 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:15.275419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8beca6ba-32d9-4136-af4f-cab3c5fcfe31 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:15.277891 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8beca6ba-32d9-4136-af4f-cab3c5fcfe31 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:15.293813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1059/2129] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:14 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:46:15.350468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:15.354022 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:15.510420 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fec9e0-0cee-4675-8157-d30f919a9403 service placement] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:15.521407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:15.523625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:15.524167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:15.524656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] PATH_INFO: `/v3/projects/c0151d58abd047a692f5cc4e7d05bea5` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:15.526980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0151d58abd047a692f5cc4e7d05bea5)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:15.531503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:15.668357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0151d58abd047a692f5cc4e7d05bea5', 'request_id': 'req-9380827e-8a60-46df-bb61-1a73d9ba7b23'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:15.682617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0151d58abd047a692f5cc4e7d05bea5', 'request_id': 'req-9380827e-8a60-46df-bb61-1a73d9ba7b23'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:15.696935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0151d58abd047a692f5cc4e7d05bea5', 'request_id': 'req-9380827e-8a60-46df-bb61-1a73d9ba7b23'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:15.722707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] The token cache is being invalidate because project c0151d58abd047a692f5cc4e7d05bea5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:46:15.723317 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:15.724518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9380827e-8a60-46df-bb61-1a73d9ba7b23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:46:15.728382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1071/2130] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:46:15 2022] DELETE /identity/v3/projects/c0151d58abd047a692f5cc4e7d05bea5 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:46:15.735114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b35bb-2a95-4aa4-b6bc-89043cc5e56b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:15.736057 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b35bb-2a95-4aa4-b6bc-89043cc5e56b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:15.736582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b35bb-2a95-4aa4-b6bc-89043cc5e56b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:15.766627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d23b35bb-2a95-4aa4-b6bc-89043cc5e56b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:15.768281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d23b35bb-2a95-4aa4-b6bc-89043cc5e56b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:15.958530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1060/2131] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:46:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:46:15.968578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6b05f5-daf3-4294-81aa-92b700e69fd8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:15.969231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6b05f5-daf3-4294-81aa-92b700e69fd8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:15.970057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6b05f5-daf3-4294-81aa-92b700e69fd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.017679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-4a6b05f5-daf3-4294-81aa-92b700e69fd8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:16.020225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4a6b05f5-daf3-4294-81aa-92b700e69fd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:16.035248 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1072/2132] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:46:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:46:16.041534 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23b35bb-2a95-4aa4-b6bc-89043cc5e56b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:16.053240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23b35bb-2a95-4aa4-b6bc-89043cc5e56b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:16.059244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:16.059834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:16.060117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:16.060392 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.061068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:46:16.061369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:16.063059 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:16.089598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1061/2133] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:46:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:46:16.090741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6b05f5-daf3-4294-81aa-92b700e69fd8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:16.102995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6b05f5-daf3-4294-81aa-92b700e69fd8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:16.109169 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f2a833-e03f-4f0d-a760-d4e4b1d74086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:16.109769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f2a833-e03f-4f0d-a760-d4e4b1d74086 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:16.110138 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f2a833-e03f-4f0d-a760-d4e4b1d74086 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:16.110541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f2a833-e03f-4f0d-a760-d4e4b1d74086 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.111205 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f2a833-e03f-4f0d-a760-d4e4b1d74086 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:16.113359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f2a833-e03f-4f0d-a760-d4e4b1d74086 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:16.155680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1073/2134] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:46:16 2022] POST /identity/v3/projects => generated 373 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:46:16.160097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:16.173320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db207c-40ec-477c-a188-4768c02decb3 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:16.241740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:16.265321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4d3db9-5ebd-43e8-b20e-d58bf6e1909e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:16.265848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d3db9-5ebd-43e8-b20e-d58bf6e1909e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:16.266122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d3db9-5ebd-43e8-b20e-d58bf6e1909e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:16.266396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d3db9-5ebd-43e8-b20e-d58bf6e1909e admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.267088 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4d3db9-5ebd-43e8-b20e-d58bf6e1909e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:16.269243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4d3db9-5ebd-43e8-b20e-d58bf6e1909e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:16.295544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1062/2135] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:46:16 2022] POST /identity/v3/users => generated 287 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:46:16.299855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f2a833-e03f-4f0d-a760-d4e4b1d74086 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:16.312094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f2a833-e03f-4f0d-a760-d4e4b1d74086 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:16.321601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ed323-48fe-4d34-925f-31dbee09c5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:16.322672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ed323-48fe-4d34-925f-31dbee09c5d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:16.323174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ed323-48fe-4d34-925f-31dbee09c5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:16.323663 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ed323-48fe-4d34-925f-31dbee09c5d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.324906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337ed323-48fe-4d34-925f-31dbee09c5d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:16.328188 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337ed323-48fe-4d34-925f-31dbee09c5d3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:16.354559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1074/2136] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:46:16 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:46:16.358591 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4d3db9-5ebd-43e8-b20e-d58bf6e1909e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:16.383930 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4d3db9-5ebd-43e8-b20e-d58bf6e1909e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:16.392834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5d3e35-04cd-4e36-aa4f-285284a3389d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:16.394230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d3e35-04cd-4e36-aa4f-285284a3389d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:16.394777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d3e35-04cd-4e36-aa4f-285284a3389d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:16.395274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d3e35-04cd-4e36-aa4f-285284a3389d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.396491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5d3e35-04cd-4e36-aa4f-285284a3389d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:16.400793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5d3e35-04cd-4e36-aa4f-285284a3389d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:16.419026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1063/2137] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:46:16 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:46:16.426344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ed323-48fe-4d34-925f-31dbee09c5d3 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:16.448140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ed323-48fe-4d34-925f-31dbee09c5d3 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:16.456479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:16.457648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:16.458144 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:16.458694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] PATH_INFO: `/v3/projects/82d7723ef95b4c3faeac9835eeb233a5/users/a48ed2d9ded741ca91afc1fbbd9c608f/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.488554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82d7723ef95b4c3faeac9835eeb233a5, user_id=a48ed2d9ded741ca91afc1fbbd9c608f, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:16.494561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:16.519092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1075/2138] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:46:16 2022] PUT /identity/v3/projects/82d7723ef95b4c3faeac9835eeb233a5/users/a48ed2d9ded741ca91afc1fbbd9c608f/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:46:16.527843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464b368-6d22-4de6-ac6d-404ac6629308 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:16.528227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464b368-6d22-4de6-ac6d-404ac6629308 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:16.528579 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464b368-6d22-4de6-ac6d-404ac6629308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.550467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5464b368-6d22-4de6-ac6d-404ac6629308 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:16.551766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5464b368-6d22-4de6-ac6d-404ac6629308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:16.624932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1064/2139] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:46:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:46:16.635997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:16.637838 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:16.658879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:16.715165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72349299-17ea-4ebe-ac9c-69d4f8e6e4a4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:16.719710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992faa79-2581-434c-971e-33445309c1c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:16.720520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992faa79-2581-434c-971e-33445309c1c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:16.720867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992faa79-2581-434c-971e-33445309c1c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:16.721187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992faa79-2581-434c-971e-33445309c1c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:16.733423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992faa79-2581-434c-971e-33445309c1c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:16.735012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992faa79-2581-434c-971e-33445309c1c3 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:16.747156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1076/2140] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:18.291046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464b368-6d22-4de6-ac6d-404ac6629308 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:18.310678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464b368-6d22-4de6-ac6d-404ac6629308 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:18.326319 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:18.329487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:18.335398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:18.336050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:18.338939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:18.356635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:18.374658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1065/2141] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:23.828736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c3f426-bf58-490c-8873-f63b312cc0dc None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:23.829556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c3f426-bf58-490c-8873-f63b312cc0dc None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:23.829556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c3f426-bf58-490c-8873-f63b312cc0dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:23.873394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-09c3f426-bf58-490c-8873-f63b312cc0dc None None] MFA Rules not processed for user `a48ed2d9ded741ca91afc1fbbd9c608f`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:23.875227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-09c3f426-bf58-490c-8873-f63b312cc0dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:24.002462 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1077/2142] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:46:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:46:24.023767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:24.027302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:24.140173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25187126-ebce-4431-ac8e-65233c2a9644 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:24.150067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e7c469-a8b4-4a39-83c9-4579760970c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:24.150959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e7c469-a8b4-4a39-83c9-4579760970c6 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:24.151443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e7c469-a8b4-4a39-83c9-4579760970c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:24.151894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e7c469-a8b4-4a39-83c9-4579760970c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:24.165485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e7c469-a8b4-4a39-83c9-4579760970c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:24.167540 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e7c469-a8b4-4a39-83c9-4579760970c6 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:24.182839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1066/2143] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:26.360447 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bbc573-dd86-4d0a-98e6-e33efb5f45b8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:26.361305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bbc573-dd86-4d0a-98e6-e33efb5f45b8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:26.361452 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bbc573-dd86-4d0a-98e6-e33efb5f45b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:26.398045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-09bbc573-dd86-4d0a-98e6-e33efb5f45b8 None None] MFA Rules not processed for user `a48ed2d9ded741ca91afc1fbbd9c608f`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:26.399439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-09bbc573-dd86-4d0a-98e6-e33efb5f45b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:26.409786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1078/2144] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:46:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:46:27.560560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e7c469-a8b4-4a39-83c9-4579760970c6 service nova] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:27.587696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e7c469-a8b4-4a39-83c9-4579760970c6 service nova] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:27.602289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:27.603606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:27.604090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:27.604600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:27.607842 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:27.633810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:27.637295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:27.673143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1067/2145] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:31.973924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166630f0-5b6a-4da1-b640-6ed9c3c5db05 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:31.975300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166630f0-5b6a-4da1-b640-6ed9c3c5db05 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:31.975300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166630f0-5b6a-4da1-b640-6ed9c3c5db05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:32.000772 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-166630f0-5b6a-4da1-b640-6ed9c3c5db05 None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:32.002713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-166630f0-5b6a-4da1-b640-6ed9c3c5db05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:32.086508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1079/2146] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:46:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:46:32.182367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:32.184880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:32.252519 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:32.325362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e35c077-1e02-4b91-b567-aa85cb396ac6 service nova] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:32.333132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9aa597-ceb0-4dcc-b717-6d85281fa42f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:32.333968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9aa597-ceb0-4dcc-b717-6d85281fa42f service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:32.334553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9aa597-ceb0-4dcc-b717-6d85281fa42f service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:32.334994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9aa597-ceb0-4dcc-b717-6d85281fa42f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:32.348260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9aa597-ceb0-4dcc-b717-6d85281fa42f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:32.350540 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9aa597-ceb0-4dcc-b717-6d85281fa42f service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:32.367716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1068/2147] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:46:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 00:46:42.072680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=79 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:42.356801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=22 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:46:45.900398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166630f0-5b6a-4da1-b640-6ed9c3c5db05 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:45.919422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166630f0-5b6a-4da1-b640-6ed9c3c5db05 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:45.926173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08f89ea-968d-412c-97bc-51324e3729c4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:45.926981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08f89ea-968d-412c-97bc-51324e3729c4 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:45.927258 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08f89ea-968d-412c-97bc-51324e3729c4 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:45.927451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08f89ea-968d-412c-97bc-51324e3729c4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:45.929832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a08f89ea-968d-412c-97bc-51324e3729c4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:45.945946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08f89ea-968d-412c-97bc-51324e3729c4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:45.948802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08f89ea-968d-412c-97bc-51324e3729c4 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:45.961315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1080/2148] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:51.132542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122425e-748c-4756-95fc-6633521d2990 None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.133135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122425e-748c-4756-95fc-6633521d2990 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.133391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122425e-748c-4756-95fc-6633521d2990 None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.136012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1069/2149] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:46:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:46:51.142908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb12bab-bd8d-4db9-8313-ca2ba0994c16 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.143524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb12bab-bd8d-4db9-8313-ca2ba0994c16 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.143840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb12bab-bd8d-4db9-8313-ca2ba0994c16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.172048 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9fb12bab-bd8d-4db9-8313-ca2ba0994c16 None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:51.173543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9fb12bab-bd8d-4db9-8313-ca2ba0994c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:51.184152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1081/2150] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:46:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:46:51.199620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c122425e-748c-4756-95fc-6633521d2990 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:51.201834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c122425e-748c-4756-95fc-6633521d2990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:51.288427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c122425e-748c-4756-95fc-6633521d2990 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:51.293849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720f3b9-dc44-4f13-9139-e6ef6efec32c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e3f6be584e34bfb9e430c7756d071d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:51.295862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f3b9-dc44-4f13-9139-e6ef6efec32c service glance] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.296457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f3b9-dc44-4f13-9139-e6ef6efec32c service glance] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.296949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f3b9-dc44-4f13-9139-e6ef6efec32c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.317027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720f3b9-dc44-4f13-9139-e6ef6efec32c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:51.320407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720f3b9-dc44-4f13-9139-e6ef6efec32c service glance] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:51.339370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1070/2151] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:46:51.424718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6c2e5c-e143-477c-86f1-122360b51fba None None] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.425041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6c2e5c-e143-477c-86f1-122360b51fba None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.425276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6c2e5c-e143-477c-86f1-122360b51fba None None] PATH_INFO: `/` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.426719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1082/2152] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:46:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:46:51.432888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248fa919-3a8e-4c8f-9929-9ae2a53b177e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.433223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248fa919-3a8e-4c8f-9929-9ae2a53b177e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.433559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248fa919-3a8e-4c8f-9929-9ae2a53b177e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.457744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-248fa919-3a8e-4c8f-9929-9ae2a53b177e None None] MFA Rules not processed for user `5756f4ff604a413b8af1f12c82cc4870`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:51.459672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-248fa919-3a8e-4c8f-9929-9ae2a53b177e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:51.466704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1071/2153] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:46:51 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:46:51.483515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6c2e5c-e143-477c-86f1-122360b51fba None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:51.495817 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6c2e5c-e143-477c-86f1-122360b51fba None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:51.499683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd88157-1a42-4c27-a7ac-2f3c9bc9baca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:51.500306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd88157-1a42-4c27-a7ac-2f3c9bc9baca service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.500598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd88157-1a42-4c27-a7ac-2f3c9bc9baca service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.500918 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd88157-1a42-4c27-a7ac-2f3c9bc9baca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.510037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd88157-1a42-4c27-a7ac-2f3c9bc9baca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:51.511556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd88157-1a42-4c27-a7ac-2f3c9bc9baca service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:51.521438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1083/2154] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:46:51.616296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efda6d45-b329-47a3-9289-e13d9c268ddf None None] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.616668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efda6d45-b329-47a3-9289-e13d9c268ddf None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.617036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efda6d45-b329-47a3-9289-e13d9c268ddf None None] PATH_INFO: `/` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.618731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1072/2155] 10.222.0.26 () {54 vars in 921 bytes} [Wed Oct 19 00:46:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 00:46:51.625878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502a3f96-671a-46b0-8d53-9d40a00f13df None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.626379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502a3f96-671a-46b0-8d53-9d40a00f13df None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.626897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502a3f96-671a-46b0-8d53-9d40a00f13df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.662545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-502a3f96-671a-46b0-8d53-9d40a00f13df None None] MFA Rules not processed for user `41009aec3b1843aa80323c374cc01461`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:51.664698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-502a3f96-671a-46b0-8d53-9d40a00f13df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:51.681892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1084/2156] 10.222.0.26 () {58 vars in 1018 bytes} [Wed Oct 19 00:46:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:46:51.693675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efda6d45-b329-47a3-9289-e13d9c268ddf None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:51.695553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-efda6d45-b329-47a3-9289-e13d9c268ddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:51.710653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efda6d45-b329-47a3-9289-e13d9c268ddf None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:51.714822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a35b324-085d-4220-aa15-0ebb63ff85f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:51.715580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35b324-085d-4220-aa15-0ebb63ff85f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:51.715867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35b324-085d-4220-aa15-0ebb63ff85f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:51.716141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35b324-085d-4220-aa15-0ebb63ff85f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:51.727966 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a35b324-085d-4220-aa15-0ebb63ff85f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:51.729445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a35b324-085d-4220-aa15-0ebb63ff85f6 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:51.743143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1073/2157] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:46:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:46:54.053943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ad211-c729-4aa7-9d32-dc07ce0727db None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:54.054595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ad211-c729-4aa7-9d32-dc07ce0727db None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:54.054595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ad211-c729-4aa7-9d32-dc07ce0727db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:54.078190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f84ad211-c729-4aa7-9d32-dc07ce0727db None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:46:54.079566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f84ad211-c729-4aa7-9d32-dc07ce0727db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:46:54.087438 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1085/2158] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:46:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:46:54.101756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a35b324-085d-4220-aa15-0ebb63ff85f6 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:46:54.115655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a35b324-085d-4220-aa15-0ebb63ff85f6 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:46:54.120866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724b0bd-9883-4534-96ff-d5b4f9cf5b77 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:46:54.121491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724b0bd-9883-4534-96ff-d5b4f9cf5b77 service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:46:54.121773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724b0bd-9883-4534-96ff-d5b4f9cf5b77 service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:46:54.122047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724b0bd-9883-4534-96ff-d5b4f9cf5b77 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:46:54.132610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6724b0bd-9883-4534-96ff-d5b4f9cf5b77 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:46:54.134235 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6724b0bd-9883-4534-96ff-d5b4f9cf5b77 service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:46:54.144419 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1074/2159] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:46:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:47:04.072873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=21 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:47:04.140126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=41 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:47:11.986181 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f11767-550b-4187-b92a-d868fc4247b3 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:11.987617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f11767-550b-4187-b92a-d868fc4247b3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:11.987617 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f11767-550b-4187-b92a-d868fc4247b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:12.007846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-36f11767-550b-4187-b92a-d868fc4247b3 None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:47:12.009106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-36f11767-550b-4187-b92a-d868fc4247b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:12.018428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1086/2160] 10.222.0.26 () {58 vars in 967 bytes} [Wed Oct 19 00:47:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:47:12.031424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724b0bd-9883-4534-96ff-d5b4f9cf5b77 service swift] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:12.044944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724b0bd-9883-4534-96ff-d5b4f9cf5b77 service swift] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:12.050940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cd5b73-9a90-450a-8f8c-35540829d44d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:12.051967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cd5b73-9a90-450a-8f8c-35540829d44d service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:12.052359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cd5b73-9a90-450a-8f8c-35540829d44d service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:12.052766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cd5b73-9a90-450a-8f8c-35540829d44d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:12.063417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cd5b73-9a90-450a-8f8c-35540829d44d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:12.065388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8cd5b73-9a90-450a-8f8c-35540829d44d service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:12.076017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1075/2161] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:47:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 00:47:16.627056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ada113a-3876-4c8b-9bdb-8b979ee82ded None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:16.627932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ada113a-3876-4c8b-9bdb-8b979ee82ded None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:16.627932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ada113a-3876-4c8b-9bdb-8b979ee82ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:16.650016 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5ada113a-3876-4c8b-9bdb-8b979ee82ded None None] MFA Rules not processed for user `f738e67d8c2340028da1254b60d7810e`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:47:16.651290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5ada113a-3876-4c8b-9bdb-8b979ee82ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:16.730098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1087/2162] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:47:16 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:47:22.071961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:47:26.723726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=28 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:47:46.576048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cd5b73-9a90-450a-8f8c-35540829d44d service swift] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:46.589073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cd5b73-9a90-450a-8f8c-35540829d44d service swift] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:46.593647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:46.594794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:46.595354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:46.595875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:46.598765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:46.614848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:46.616317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:46.628002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1076/2163] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:47:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:47:52.345193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ada113a-3876-4c8b-9bdb-8b979ee82ded None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:52.351203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5ada113a-3876-4c8b-9bdb-8b979ee82ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:52.379628 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ada113a-3876-4c8b-9bdb-8b979ee82ded None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:52.387945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:52.389288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:52.389698 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:52.390098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] PATH_INFO: `/v3/users/6f8725c4d05b4dd79b05b43fba15980b` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:52.392301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f8725c4d05b4dd79b05b43fba15980b)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:52.396544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:52.471298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f8725c4d05b4dd79b05b43fba15980b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.480937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f8725c4d05b4dd79b05b43fba15980b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.481909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f8725c4d05b4dd79b05b43fba15980b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.482933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f8725c4d05b4dd79b05b43fba15980b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.493167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-265afbbf-5a8c-4b17-92e8-e908bc690814 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f8725c4d05b4dd79b05b43fba15980b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.519108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1088/2164] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:47:52 2022] DELETE /identity/v3/users/6f8725c4d05b4dd79b05b43fba15980b => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:47:52.748238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:52.760665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2019da-4083-489e-8d2d-c3b91a4f0067 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:52.835225 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:52.835931 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:52.836421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:52.836850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] PATH_INFO: `/v3/projects/0b35c2038f6b4d0a9012b004f1416c00` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:52.838429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b35c2038f6b4d0a9012b004f1416c00)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:52.841051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:52.943394 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b35c2038f6b4d0a9012b004f1416c00', 'request_id': 'req-327f7666-73e5-411b-bbc1-2cd883f0b772'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.953821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b35c2038f6b4d0a9012b004f1416c00', 'request_id': 'req-327f7666-73e5-411b-bbc1-2cd883f0b772'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.961627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b35c2038f6b4d0a9012b004f1416c00', 'request_id': 'req-327f7666-73e5-411b-bbc1-2cd883f0b772'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.981770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] The token cache is being invalidate because project 0b35c2038f6b4d0a9012b004f1416c00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:47:52.982421 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.983675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-327f7666-73e5-411b-bbc1-2cd883f0b772 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:47:52.986367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1077/2165] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:47:52 2022] DELETE /identity/v3/projects/0b35c2038f6b4d0a9012b004f1416c00 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:47:52.992579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae871d6f-2192-4437-be9d-80ea67ed96e3 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:52.992872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae871d6f-2192-4437-be9d-80ea67ed96e3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:52.993124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae871d6f-2192-4437-be9d-80ea67ed96e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.017094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ae871d6f-2192-4437-be9d-80ea67ed96e3 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:47:53.018377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ae871d6f-2192-4437-be9d-80ea67ed96e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:53.132162 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1089/2166] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:47:53.142732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d837f3-b0fd-4901-95e1-d2ed2fddf92a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.143061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d837f3-b0fd-4901-95e1-d2ed2fddf92a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.145592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d837f3-b0fd-4901-95e1-d2ed2fddf92a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.176100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-33d837f3-b0fd-4901-95e1-d2ed2fddf92a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:47:53.177643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-33d837f3-b0fd-4901-95e1-d2ed2fddf92a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:53.189799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1078/2167] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:47:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:47:53.191878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae871d6f-2192-4437-be9d-80ea67ed96e3 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:53.202620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae871d6f-2192-4437-be9d-80ea67ed96e3 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:53.208386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:53.209019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.209294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.209564 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.210228 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:47:53.210567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:53.212276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:53.229937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1090/2168] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:47:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:47:53.234622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d837f3-b0fd-4901-95e1-d2ed2fddf92a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:53.245004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d837f3-b0fd-4901-95e1-d2ed2fddf92a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:53.251349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07629a-610a-4e81-adf2-afa907e11894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:53.252456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07629a-610a-4e81-adf2-afa907e11894 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.253217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07629a-610a-4e81-adf2-afa907e11894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.253504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07629a-610a-4e81-adf2-afa907e11894 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.254827 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff07629a-610a-4e81-adf2-afa907e11894 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:53.258975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff07629a-610a-4e81-adf2-afa907e11894 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:53.307434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1079/2169] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:47:53 2022] POST /identity/v3/projects => generated 367 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:47:53.312582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:53.325037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f43667-3ed5-4298-ad67-8ec2092a26dd admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:53.402211 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d60b31-05d7-437b-8133-1f2c0a7a1b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:53.403339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d60b31-05d7-437b-8133-1f2c0a7a1b45 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.403694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d60b31-05d7-437b-8133-1f2c0a7a1b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.404153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d60b31-05d7-437b-8133-1f2c0a7a1b45 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.405178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d60b31-05d7-437b-8133-1f2c0a7a1b45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:53.407928 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d60b31-05d7-437b-8133-1f2c0a7a1b45 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:53.419742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:47:53.452319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1091/2170] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:47:53 2022] POST /identity/v3/users => generated 284 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:47:53.459415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07629a-610a-4e81-adf2-afa907e11894 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:53.475506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07629a-610a-4e81-adf2-afa907e11894 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:53.483009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2e538e-e687-4cf6-ab70-3e309b7581c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:53.484040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2e538e-e687-4cf6-ab70-3e309b7581c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.484541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2e538e-e687-4cf6-ab70-3e309b7581c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.485049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2e538e-e687-4cf6-ab70-3e309b7581c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.486129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2e538e-e687-4cf6-ab70-3e309b7581c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:53.489875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2e538e-e687-4cf6-ab70-3e309b7581c7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:53.506523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1080/2171] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:47:53.511842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d60b31-05d7-437b-8133-1f2c0a7a1b45 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:53.533320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d60b31-05d7-437b-8133-1f2c0a7a1b45 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:53.549163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265e7d3d-38e9-482e-ab31-9ce145b56807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:53.552944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265e7d3d-38e9-482e-ab31-9ce145b56807 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.553892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265e7d3d-38e9-482e-ab31-9ce145b56807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.555476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265e7d3d-38e9-482e-ab31-9ce145b56807 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.557242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265e7d3d-38e9-482e-ab31-9ce145b56807 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:53.561193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265e7d3d-38e9-482e-ab31-9ce145b56807 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:53.589431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1092/2172] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:47:53.596184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2e538e-e687-4cf6-ab70-3e309b7581c7 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:53.622324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2e538e-e687-4cf6-ab70-3e309b7581c7 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:53.630530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:53.631384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.631781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.632311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] PATH_INFO: `/v3/projects/f7959c8665924b50944dd1187a887794/users/9e40e7a1f297426880b52ce356b043ce/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.653543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7959c8665924b50944dd1187a887794, user_id=9e40e7a1f297426880b52ce356b043ce, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:53.657449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:53.678363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1081/2173] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:47:53 2022] PUT /identity/v3/projects/f7959c8665924b50944dd1187a887794/users/9e40e7a1f297426880b52ce356b043ce/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:47:53.694130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a48b5a-31c7-41cf-8a79-2bf0215365ca None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.694511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a48b5a-31c7-41cf-8a79-2bf0215365ca None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.694983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a48b5a-31c7-41cf-8a79-2bf0215365ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.750706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-13a48b5a-31c7-41cf-8a79-2bf0215365ca None None] MFA Rules not processed for user `9e40e7a1f297426880b52ce356b043ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:47:53.753175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-13a48b5a-31c7-41cf-8a79-2bf0215365ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:53.860081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1093/2174] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:47:53 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:47:53.875340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:53.877731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:53.897863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:47:53.954754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc359ba-765e-4e71-842c-cc9c70ad9618 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:53.959218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:53.959855 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:53.960124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:53.960631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:53.970055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:53.971843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:53.983781 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1082/2175] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:47:53 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:47:56.386344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a48b5a-31c7-41cf-8a79-2bf0215365ca None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:47:56.387912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-13a48b5a-31c7-41cf-8a79-2bf0215365ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:56.463231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a48b5a-31c7-41cf-8a79-2bf0215365ca None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:47:56.467600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:47:56.468266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:47:56.468545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:47:56.468813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:47:56.471088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:47:56.549474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:47:56.552434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:47:56.566806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1094/2176] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:47:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:01.320919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:01.323592 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:01.346049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfe85f3-2fbe-4973-8ddc-b5e42b5dc1ba service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:01.353668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:01.354782 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:01.355373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:01.355813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] PATH_INFO: `/v3/users/69fef5650fff42e4ac4ed3ca74d4b123` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:01.359666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] RBAC: Authorizing `identity:delete_user(user_id=69fef5650fff42e4ac4ed3ca74d4b123)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:01.364442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:01.426643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '69fef5650fff42e4ac4ed3ca74d4b123'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.437617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '69fef5650fff42e4ac4ed3ca74d4b123'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.438937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '69fef5650fff42e4ac4ed3ca74d4b123'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.447024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '69fef5650fff42e4ac4ed3ca74d4b123'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.460807 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e0988b31-5cb5-4284-8b47-c33834edb02a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '69fef5650fff42e4ac4ed3ca74d4b123'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.495449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1083/2177] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:01 2022] DELETE /identity/v3/users/69fef5650fff42e4ac4ed3ca74d4b123 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:01.729702 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:01.746477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7197bcc-c73a-4d68-8c1f-9d595004f8ea service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:01.766570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:01.856343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:01.857062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:01.857342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:01.857840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] PATH_INFO: `/v3/projects/37ffcffcae9b41bbb2d13adc49cb99df` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:01.859391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37ffcffcae9b41bbb2d13adc49cb99df)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:01.861834 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:01.933876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37ffcffcae9b41bbb2d13adc49cb99df', 'request_id': 'req-37bdd677-20f6-4f3d-93dc-c96fe8212e57'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.944375 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37ffcffcae9b41bbb2d13adc49cb99df', 'request_id': 'req-37bdd677-20f6-4f3d-93dc-c96fe8212e57'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.954645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37ffcffcae9b41bbb2d13adc49cb99df', 'request_id': 'req-37bdd677-20f6-4f3d-93dc-c96fe8212e57'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.969543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] The token cache is being invalidate because project 37ffcffcae9b41bbb2d13adc49cb99df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:01.969816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.970519 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-37bdd677-20f6-4f3d-93dc-c96fe8212e57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:01.974871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1095/2178] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:01 2022] DELETE /identity/v3/projects/37ffcffcae9b41bbb2d13adc49cb99df => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:01.980373 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc69f3c-aa7c-4c06-97a1-3fdd732818f9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:01.980661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc69f3c-aa7c-4c06-97a1-3fdd732818f9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:01.980953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc69f3c-aa7c-4c06-97a1-3fdd732818f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.004273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-3bc69f3c-aa7c-4c06-97a1-3fdd732818f9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:02.005711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3bc69f3c-aa7c-4c06-97a1-3fdd732818f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:02.083210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1084/2179] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:02.092066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71f5a44-2c12-445d-a33c-d31e183e6ebf None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.092483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71f5a44-2c12-445d-a33c-d31e183e6ebf None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.093081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71f5a44-2c12-445d-a33c-d31e183e6ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.118682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c71f5a44-2c12-445d-a33c-d31e183e6ebf None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:02.120260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c71f5a44-2c12-445d-a33c-d31e183e6ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:02.130415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1096/2180] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:02.135168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc69f3c-aa7c-4c06-97a1-3fdd732818f9 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:02.146969 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc69f3c-aa7c-4c06-97a1-3fdd732818f9 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:02.152063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:02.152691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.152972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.153275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.153949 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:02.154252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:02.155958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:02.172412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1085/2181] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:48:02.178211 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71f5a44-2c12-445d-a33c-d31e183e6ebf None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:02.192578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71f5a44-2c12-445d-a33c-d31e183e6ebf None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:02.199160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36525e6-af13-4410-8861-934b62624bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:02.199884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36525e6-af13-4410-8861-934b62624bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.200298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36525e6-af13-4410-8861-934b62624bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.200620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36525e6-af13-4410-8861-934b62624bfc admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.201573 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36525e6-af13-4410-8861-934b62624bfc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:02.204249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36525e6-af13-4410-8861-934b62624bfc admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:02.243869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1097/2182] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:02 2022] POST /identity/v3/projects => generated 373 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:02.248328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:02.267056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a618409-f22e-4da5-8331-5e2dbfdbed2e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:02.282968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:02.350686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df309f9-e38c-4d1b-8fe3-6c1f23ab3c3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:02.351536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df309f9-e38c-4d1b-8fe3-6c1f23ab3c3e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.351708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df309f9-e38c-4d1b-8fe3-6c1f23ab3c3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.351988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df309f9-e38c-4d1b-8fe3-6c1f23ab3c3e admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.352938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df309f9-e38c-4d1b-8fe3-6c1f23ab3c3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:02.355475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df309f9-e38c-4d1b-8fe3-6c1f23ab3c3e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:02.384313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1086/2183] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:02 2022] POST /identity/v3/users => generated 287 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:02.389167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36525e6-af13-4410-8861-934b62624bfc admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:02.400612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36525e6-af13-4410-8861-934b62624bfc admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:02.406368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca25977-6c74-4e41-ad3f-292743d16a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:02.407062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca25977-6c74-4e41-ad3f-292743d16a07 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.407350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca25977-6c74-4e41-ad3f-292743d16a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.407781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca25977-6c74-4e41-ad3f-292743d16a07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.408485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca25977-6c74-4e41-ad3f-292743d16a07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:02.410734 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca25977-6c74-4e41-ad3f-292743d16a07 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:02.421971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1098/2184] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:02 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:02.427797 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df309f9-e38c-4d1b-8fe3-6c1f23ab3c3e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:02.439066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df309f9-e38c-4d1b-8fe3-6c1f23ab3c3e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:02.444580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d821b4fa-da63-4b99-b2f4-2e5b22e763e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:02.445206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d821b4fa-da63-4b99-b2f4-2e5b22e763e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.445516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d821b4fa-da63-4b99-b2f4-2e5b22e763e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.445788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d821b4fa-da63-4b99-b2f4-2e5b22e763e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.446458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d821b4fa-da63-4b99-b2f4-2e5b22e763e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:02.448143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d821b4fa-da63-4b99-b2f4-2e5b22e763e7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:02.461777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1087/2185] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:02 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:02.468389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca25977-6c74-4e41-ad3f-292743d16a07 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:02.481068 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca25977-6c74-4e41-ad3f-292743d16a07 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:02.486342 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:02.487102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.487540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.488018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] PATH_INFO: `/v3/projects/ec9886511380430d8ca93cb3fdb85293/users/768941ad9b904febaf4cd2c7b162f799/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.506736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec9886511380430d8ca93cb3fdb85293, user_id=768941ad9b904febaf4cd2c7b162f799, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:02.509533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:02.526129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1099/2186] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:02 2022] PUT /identity/v3/projects/ec9886511380430d8ca93cb3fdb85293/users/768941ad9b904febaf4cd2c7b162f799/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:02.545885 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f76b884-1148-4c34-919c-235858384fb1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.546417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f76b884-1148-4c34-919c-235858384fb1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.546947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f76b884-1148-4c34-919c-235858384fb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.586318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7f76b884-1148-4c34-919c-235858384fb1 None None] MFA Rules not processed for user `768941ad9b904febaf4cd2c7b162f799`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:02.588246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7f76b884-1148-4c34-919c-235858384fb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:02.678769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1088/2187] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:02.693711 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:02.696574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:02.765619 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:02.803790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7868a77f-e117-480f-8ab9-ac151c55151c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:02.810904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384914ff-3472-4752-b815-3b61973ad2bd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:02.811546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384914ff-3472-4752-b815-3b61973ad2bd service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:02.811919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384914ff-3472-4752-b815-3b61973ad2bd service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:02.812259 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384914ff-3472-4752-b815-3b61973ad2bd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:02.823292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384914ff-3472-4752-b815-3b61973ad2bd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:02.824930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384914ff-3472-4752-b815-3b61973ad2bd service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:02.836182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1100/2188] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:02 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:04.172416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f76b884-1148-4c34-919c-235858384fb1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:04.173972 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7f76b884-1148-4c34-919c-235858384fb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:04.255400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f76b884-1148-4c34-919c-235858384fb1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:04.260623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:04.261433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:04.261824 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:04.262219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] PATH_INFO: `/v3/users/9e40e7a1f297426880b52ce356b043ce` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:04.263780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e40e7a1f297426880b52ce356b043ce)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:04.265886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:04.300517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:04.328086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e40e7a1f297426880b52ce356b043ce'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:04.337307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e40e7a1f297426880b52ce356b043ce'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:04.338065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e40e7a1f297426880b52ce356b043ce'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:04.338726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e40e7a1f297426880b52ce356b043ce'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:04.348815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e40e7a1f297426880b52ce356b043ce'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:04.376303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1089/2189] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:04 2022] DELETE /identity/v3/users/9e40e7a1f297426880b52ce356b043ce => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:04.383394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9e1c8c-02c5-480a-87a6-7db96d1ee5bd None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:04.384015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9e1c8c-02c5-480a-87a6-7db96d1ee5bd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:04.384015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9e1c8c-02c5-480a-87a6-7db96d1ee5bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:04.411085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-cf9e1c8c-02c5-480a-87a6-7db96d1ee5bd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:04.412337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cf9e1c8c-02c5-480a-87a6-7db96d1ee5bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:04.501321 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1101/2190] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:04.519003 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:04.521243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:04.637239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73e0a29-03af-4083-b716-7a31d627e769 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:04.644296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b0c61a-a6cc-4898-9eb8-323f07b33d0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:04.645548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b0c61a-a6cc-4898-9eb8-323f07b33d0e service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:04.646017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b0c61a-a6cc-4898-9eb8-323f07b33d0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:04.646478 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b0c61a-a6cc-4898-9eb8-323f07b33d0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:04.664303 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b0c61a-a6cc-4898-9eb8-323f07b33d0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:04.666974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b0c61a-a6cc-4898-9eb8-323f07b33d0e service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:04.687988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1090/2191] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:05.119466 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9e1c8c-02c5-480a-87a6-7db96d1ee5bd None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:05.122065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-cf9e1c8c-02c5-480a-87a6-7db96d1ee5bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:05.139524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9e1c8c-02c5-480a-87a6-7db96d1ee5bd None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:05.148515 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:05.149631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.150206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.150783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] PATH_INFO: `/v3/projects/f7959c8665924b50944dd1187a887794` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.153056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7959c8665924b50944dd1187a887794)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:05.156903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:05.244155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7959c8665924b50944dd1187a887794', 'request_id': 'req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:05.251145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7959c8665924b50944dd1187a887794', 'request_id': 'req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:05.260834 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7959c8665924b50944dd1187a887794', 'request_id': 'req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:05.275980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] The token cache is being invalidate because project f7959c8665924b50944dd1187a887794 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:05.276380 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:05.277234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-9f75a5fb-9fd9-4a6d-b36f-ff7293dc36cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:05.279028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1102/2192] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:05 2022] DELETE /identity/v3/projects/f7959c8665924b50944dd1187a887794 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:05.285839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff90af0-aaec-4f73-81af-ff62839ba565 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.286627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff90af0-aaec-4f73-81af-ff62839ba565 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.286860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff90af0-aaec-4f73-81af-ff62839ba565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.321541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5ff90af0-aaec-4f73-81af-ff62839ba565 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:05.323484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5ff90af0-aaec-4f73-81af-ff62839ba565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:05.399370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1091/2193] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:05.407789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c88bd33-a8c4-49f9-888e-dc0260998c62 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.408490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c88bd33-a8c4-49f9-888e-dc0260998c62 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.408490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c88bd33-a8c4-49f9-888e-dc0260998c62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.431105 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-4c88bd33-a8c4-49f9-888e-dc0260998c62 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:05.432367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4c88bd33-a8c4-49f9-888e-dc0260998c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:05.442127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1103/2194] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:05.446777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff90af0-aaec-4f73-81af-ff62839ba565 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:05.458091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff90af0-aaec-4f73-81af-ff62839ba565 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:05.464246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:05.464988 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.465305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.465614 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.466364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:05.466751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:05.468599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:05.483377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1092/2195] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:48:05.490720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c88bd33-a8c4-49f9-888e-dc0260998c62 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:05.500970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c88bd33-a8c4-49f9-888e-dc0260998c62 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:05.508045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040cf943-d605-4c44-a0de-f2782422e468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:05.508734 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040cf943-d605-4c44-a0de-f2782422e468 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.509031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040cf943-d605-4c44-a0de-f2782422e468 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.509381 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040cf943-d605-4c44-a0de-f2782422e468 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.510141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040cf943-d605-4c44-a0de-f2782422e468 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:05.512668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040cf943-d605-4c44-a0de-f2782422e468 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:05.550130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1104/2196] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:05 2022] POST /identity/v3/projects => generated 399 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:05.555744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:05.568454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11453497-acf6-49dd-8dbd-8d749c6ff09a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:05.610564 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:05.649123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c208896-031a-4988-b2ce-8fb8d6f01d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:05.649744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c208896-031a-4988-b2ce-8fb8d6f01d1b admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.650024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c208896-031a-4988-b2ce-8fb8d6f01d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.650304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c208896-031a-4988-b2ce-8fb8d6f01d1b admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.650995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c208896-031a-4988-b2ce-8fb8d6f01d1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:05.653005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c208896-031a-4988-b2ce-8fb8d6f01d1b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:05.681531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1093/2197] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:05 2022] POST /identity/v3/users => generated 300 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:05.686371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040cf943-d605-4c44-a0de-f2782422e468 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:05.700781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040cf943-d605-4c44-a0de-f2782422e468 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:05.709389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828ba52c-9e90-440f-8ecd-ecbd1668a57d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:05.710294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828ba52c-9e90-440f-8ecd-ecbd1668a57d admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.710771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828ba52c-9e90-440f-8ecd-ecbd1668a57d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.711304 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828ba52c-9e90-440f-8ecd-ecbd1668a57d admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.712447 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828ba52c-9e90-440f-8ecd-ecbd1668a57d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:05.715485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828ba52c-9e90-440f-8ecd-ecbd1668a57d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:05.735366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1105/2198] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:05 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:05.740131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c208896-031a-4988-b2ce-8fb8d6f01d1b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:05.751725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c208896-031a-4988-b2ce-8fb8d6f01d1b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:05.757627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:05.758318 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.758568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.758846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.759509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:05.761153 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:05.773595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1094/2199] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:05.779719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828ba52c-9e90-440f-8ecd-ecbd1668a57d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:05.791634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828ba52c-9e90-440f-8ecd-ecbd1668a57d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:05.796937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac430b9d-a1cf-4e73-b9cc-a35ab0cb7382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:05.797739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac430b9d-a1cf-4e73-b9cc-a35ab0cb7382 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.798078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac430b9d-a1cf-4e73-b9cc-a35ab0cb7382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.798413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac430b9d-a1cf-4e73-b9cc-a35ab0cb7382 admin admin] PATH_INFO: `/v3/projects/72832e3d4e51418bae334ade727d8446/users/cb60e6e7ccb44422b60640c0b3ebdc9a/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.814865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:05.822701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac430b9d-a1cf-4e73-b9cc-a35ab0cb7382 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72832e3d4e51418bae334ade727d8446, user_id=cb60e6e7ccb44422b60640c0b3ebdc9a, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:05.828652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac430b9d-a1cf-4e73-b9cc-a35ab0cb7382 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:05.858587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:05.860347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1106/2200] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:05 2022] PUT /identity/v3/projects/72832e3d4e51418bae334ade727d8446/users/cb60e6e7ccb44422b60640c0b3ebdc9a/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:05.860485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:05.871392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd8714a-9991-48d5-8916-6bf322a004fc None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.871957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd8714a-9991-48d5-8916-6bf322a004fc None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.872276 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd8714a-9991-48d5-8916-6bf322a004fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.875941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e50bf-f2a9-4329-a971-76a955771a19 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:05.907829 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8fd8714a-9991-48d5-8916-6bf322a004fc None None] MFA Rules not processed for user `cb60e6e7ccb44422b60640c0b3ebdc9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:05.910277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8fd8714a-9991-48d5-8916-6bf322a004fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:05.976479 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:05.977361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:05.977831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:05.978240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] PATH_INFO: `/v3/users/768941ad9b904febaf4cd2c7b162f799` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:05.980247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] RBAC: Authorizing `identity:delete_user(user_id=768941ad9b904febaf4cd2c7b162f799)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:05.983174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.014189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1107/2201] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:05 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:06.020030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd8714a-9991-48d5-8916-6bf322a004fc None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.037235 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd8714a-9991-48d5-8916-6bf322a004fc None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.044451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.045803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.046195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.046622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.047748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:06.051685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.060108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '768941ad9b904febaf4cd2c7b162f799'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:06.072227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '768941ad9b904febaf4cd2c7b162f799'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:06.073258 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '768941ad9b904febaf4cd2c7b162f799'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:06.074231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '768941ad9b904febaf4cd2c7b162f799'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:06.086453 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-cad27695-3b38-45ae-a08a-e3fb56a26190 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '768941ad9b904febaf4cd2c7b162f799'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:06.094401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1108/2202] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:06 2022] POST /identity/v3/projects => generated 399 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:06.100314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.102719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:06.119161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1095/2203] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:05 2022] DELETE /identity/v3/users/768941ad9b904febaf4cd2c7b162f799 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:06.128218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc1f43-96c5-4029-a1fd-fefc7ad93bd8 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.128696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc1f43-96c5-4029-a1fd-fefc7ad93bd8 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.131530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc1f43-96c5-4029-a1fd-fefc7ad93bd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.158756 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a0dc1f43-96c5-4029-a1fd-fefc7ad93bd8 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:06.161150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a0dc1f43-96c5-4029-a1fd-fefc7ad93bd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:06.202528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:06.217251 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe4d230-e84c-4c2e-b857-4c34cb08cf73 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.265538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1096/2204] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:06.279427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dc1f43-96c5-4029-a1fd-fefc7ad93bd8 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.281443 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a0dc1f43-96c5-4029-a1fd-fefc7ad93bd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:06.308812 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e34fb43-7067-4231-8190-3e029dec15b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.310306 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e34fb43-7067-4231-8190-3e029dec15b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.310809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e34fb43-7067-4231-8190-3e029dec15b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.311415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e34fb43-7067-4231-8190-3e029dec15b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.312465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e34fb43-7067-4231-8190-3e029dec15b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:06.316721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e34fb43-7067-4231-8190-3e029dec15b3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.334151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:06.355589 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1109/2205] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:06 2022] POST /identity/v3/users => generated 306 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:06.360626 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e34fb43-7067-4231-8190-3e029dec15b3 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.363438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dc1f43-96c5-4029-a1fd-fefc7ad93bd8 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.367772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c799f58a-9325-4457-9f01-fbec59063c32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.368703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c799f58a-9325-4457-9f01-fbec59063c32 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.368987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c799f58a-9325-4457-9f01-fbec59063c32 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.369251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c799f58a-9325-4457-9f01-fbec59063c32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.374921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e34fb43-7067-4231-8190-3e029dec15b3 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.381709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c799f58a-9325-4457-9f01-fbec59063c32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:06.382736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b642d-9768-4f21-9744-7d188aa54690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.383235 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c799f58a-9325-4457-9f01-fbec59063c32 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.383596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b642d-9768-4f21-9744-7d188aa54690 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.384075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b642d-9768-4f21-9744-7d188aa54690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.384763 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b642d-9768-4f21-9744-7d188aa54690 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.385647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645b642d-9768-4f21-9744-7d188aa54690 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:06.387946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645b642d-9768-4f21-9744-7d188aa54690 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.395822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1097/2206] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:06.401266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1110/2207] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:06 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:06.405992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c799f58a-9325-4457-9f01-fbec59063c32 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.417487 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c799f58a-9325-4457-9f01-fbec59063c32 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.424984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a28bef-15c8-43b3-8be9-b87766e44d47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.426098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a28bef-15c8-43b3-8be9-b87766e44d47 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.426459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a28bef-15c8-43b3-8be9-b87766e44d47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.426832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a28bef-15c8-43b3-8be9-b87766e44d47 admin admin] PATH_INFO: `/v3/projects/ef53e1b42c3840b7aaae6aea6c6fe274/users/4b79464667834bc290bbd51298654961/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.448125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a28bef-15c8-43b3-8be9-b87766e44d47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef53e1b42c3840b7aaae6aea6c6fe274, user_id=4b79464667834bc290bbd51298654961, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:06.454061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a28bef-15c8-43b3-8be9-b87766e44d47 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.476766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1098/2208] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:06 2022] PUT /identity/v3/projects/ef53e1b42c3840b7aaae6aea6c6fe274/users/4b79464667834bc290bbd51298654961/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:06.481926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b642d-9768-4f21-9744-7d188aa54690 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.497343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b642d-9768-4f21-9744-7d188aa54690 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.604755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1117f57f-f26e-48eb-b024-d18e861462b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.605494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117f57f-f26e-48eb-b024-d18e861462b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.605845 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117f57f-f26e-48eb-b024-d18e861462b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.606244 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117f57f-f26e-48eb-b024-d18e861462b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.607101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117f57f-f26e-48eb-b024-d18e861462b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:06.609568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117f57f-f26e-48eb-b024-d18e861462b3 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.627707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1111/2209] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:06 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:06.632343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a28bef-15c8-43b3-8be9-b87766e44d47 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.645100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a28bef-15c8-43b3-8be9-b87766e44d47 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.651299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.652261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.652648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.652976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] PATH_INFO: `/v3/projects/ef53e1b42c3840b7aaae6aea6c6fe274/users/4b79464667834bc290bbd51298654961/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.655726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef53e1b42c3840b7aaae6aea6c6fe274, user_id=4b79464667834bc290bbd51298654961, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:06.659544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.672131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1099/2210] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:06 2022] PUT /identity/v3/projects/ef53e1b42c3840b7aaae6aea6c6fe274/users/4b79464667834bc290bbd51298654961/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:06.684867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e3960-49e0-45d1-a693-098e59548b6f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.685186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e3960-49e0-45d1-a693-098e59548b6f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.685442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05e3960-49e0-45d1-a693-098e59548b6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.718854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-f05e3960-49e0-45d1-a693-098e59548b6f None None] MFA Rules not processed for user `4b79464667834bc290bbd51298654961`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:06.720341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f05e3960-49e0-45d1-a693-098e59548b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:06.841658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1112/2211] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:06 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:06.855480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.858298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:06.878280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05e3960-49e0-45d1-a693-098e59548b6f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:06.880029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f05e3960-49e0-45d1-a693-098e59548b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:06.957918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e905e1b-978f-4ec6-9a74-0119f93f0cba admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.962242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c6f7c7-2093-46a1-b53e-92dd283c0a20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.963041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c6f7c7-2093-46a1-b53e-92dd283c0a20 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.963322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c6f7c7-2093-46a1-b53e-92dd283c0a20 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.963600 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c6f7c7-2093-46a1-b53e-92dd283c0a20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.976325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c6f7c7-2093-46a1-b53e-92dd283c0a20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:06.977803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c6f7c7-2093-46a1-b53e-92dd283c0a20 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:06.986060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05e3960-49e0-45d1-a693-098e59548b6f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:06.990281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1100/2212] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:06 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:06.994499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:06.995543 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:06.996002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:06.996479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] PATH_INFO: `/v3/projects/ec9886511380430d8ca93cb3fdb85293` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:06.998639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec9886511380430d8ca93cb3fdb85293)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:07.002477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:07.014458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:07.092017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c6f7c7-2093-46a1-b53e-92dd283c0a20 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:07.092960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec9886511380430d8ca93cb3fdb85293', 'request_id': 'req-3a553a79-a619-444f-82d2-67618cabaebb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:07.105528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec9886511380430d8ca93cb3fdb85293', 'request_id': 'req-3a553a79-a619-444f-82d2-67618cabaebb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:07.105859 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c6f7c7-2093-46a1-b53e-92dd283c0a20 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:07.118582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec9886511380430d8ca93cb3fdb85293', 'request_id': 'req-3a553a79-a619-444f-82d2-67618cabaebb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:07.139088 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] The token cache is being invalidate because project ec9886511380430d8ca93cb3fdb85293 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:07.139568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:07.140682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-3a553a79-a619-444f-82d2-67618cabaebb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:07.143149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1113/2213] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:06 2022] DELETE /identity/v3/projects/ec9886511380430d8ca93cb3fdb85293 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:07.152288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad6212-264d-4678-a52e-22bab859deeb None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.152740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad6212-264d-4678-a52e-22bab859deeb None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.153202 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad6212-264d-4678-a52e-22bab859deeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.176955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:07.184308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-51ad6212-264d-4678-a52e-22bab859deeb None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:07.186445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-51ad6212-264d-4678-a52e-22bab859deeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:07.201275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:07.202264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.202671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.203108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.206362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:07.301541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1114/2214] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:07.309787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb8a79a-c8bc-4338-8182-f616259e3cc4 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.310093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb8a79a-c8bc-4338-8182-f616259e3cc4 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.310348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb8a79a-c8bc-4338-8182-f616259e3cc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.316844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:07.318973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:07.332356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1101/2215] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:07 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:07.343719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-afb8a79a-c8bc-4338-8182-f616259e3cc4 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:07.346771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-afb8a79a-c8bc-4338-8182-f616259e3cc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:07.362470 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1115/2216] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:07.367806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:07.386175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf28f65-fac1-4699-a76d-91a873fe9bfa service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:07.393962 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:07.394940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.395356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.395792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.396896 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:07.397416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:07.400580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:07.423937 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1102/2217] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:48:07.430786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb8a79a-c8bc-4338-8182-f616259e3cc4 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:07.446733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb8a79a-c8bc-4338-8182-f616259e3cc4 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:07.452163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab80c370-518e-4ba3-a032-b76cb39b53ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:07.452892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80c370-518e-4ba3-a032-b76cb39b53ac admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.453238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80c370-518e-4ba3-a032-b76cb39b53ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.453583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80c370-518e-4ba3-a032-b76cb39b53ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.454434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab80c370-518e-4ba3-a032-b76cb39b53ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:07.457313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab80c370-518e-4ba3-a032-b76cb39b53ac admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:07.490945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1116/2218] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:07 2022] POST /identity/v3/projects => generated 385 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:07.495625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:07.511221 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d15dbff-1d43-428a-b66a-9fdb9547b0ea admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:07.598548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83defc19-bcce-47d8-9daf-d97711d58072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:07.599524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83defc19-bcce-47d8-9daf-d97711d58072 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.599917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83defc19-bcce-47d8-9daf-d97711d58072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.600304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83defc19-bcce-47d8-9daf-d97711d58072 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.601890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83defc19-bcce-47d8-9daf-d97711d58072 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:07.603669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83defc19-bcce-47d8-9daf-d97711d58072 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:07.632149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1103/2219] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:07 2022] POST /identity/v3/users => generated 293 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:07.636522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab80c370-518e-4ba3-a032-b76cb39b53ac admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:07.649174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab80c370-518e-4ba3-a032-b76cb39b53ac admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:07.654672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb365819-c661-4d04-bd88-8869b962303b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:07.655227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb365819-c661-4d04-bd88-8869b962303b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.655696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb365819-c661-4d04-bd88-8869b962303b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb365819-c661-4d04-bd88-8869b962303b admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.656630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb365819-c661-4d04-bd88-8869b962303b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:07.658378 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb365819-c661-4d04-bd88-8869b962303b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:07.672946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1117/2220] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:07.677649 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83defc19-bcce-47d8-9daf-d97711d58072 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:07.689375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83defc19-bcce-47d8-9daf-d97711d58072 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:07.696981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f9c8a9-0f83-4a14-85a1-bd7dae159265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:07.697686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f9c8a9-0f83-4a14-85a1-bd7dae159265 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.698033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f9c8a9-0f83-4a14-85a1-bd7dae159265 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.698405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f9c8a9-0f83-4a14-85a1-bd7dae159265 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.699169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f9c8a9-0f83-4a14-85a1-bd7dae159265 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:07.700908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f9c8a9-0f83-4a14-85a1-bd7dae159265 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:07.729588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1104/2221] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:07.730889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb365819-c661-4d04-bd88-8869b962303b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:07.744331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb365819-c661-4d04-bd88-8869b962303b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:07.749391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab35515f-4e7d-42ce-9767-88646b87794b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:07.750194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35515f-4e7d-42ce-9767-88646b87794b admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.750579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35515f-4e7d-42ce-9767-88646b87794b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.750926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35515f-4e7d-42ce-9767-88646b87794b admin admin] PATH_INFO: `/v3/projects/d681ca329da2403f9710bb2cde758948/users/de163c501d594e6fb488e72f3876a82f/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.770238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab35515f-4e7d-42ce-9767-88646b87794b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d681ca329da2403f9710bb2cde758948, user_id=de163c501d594e6fb488e72f3876a82f, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:07.773701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab35515f-4e7d-42ce-9767-88646b87794b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:07.789549 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1118/2222] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:07 2022] PUT /identity/v3/projects/d681ca329da2403f9710bb2cde758948/users/de163c501d594e6fb488e72f3876a82f/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:07.802868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3719a63-3b82-4cbb-8d2f-71c743df1557 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:07.803134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3719a63-3b82-4cbb-8d2f-71c743df1557 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:07.803411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3719a63-3b82-4cbb-8d2f-71c743df1557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:07.836143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d3719a63-3b82-4cbb-8d2f-71c743df1557 None None] MFA Rules not processed for user `de163c501d594e6fb488e72f3876a82f`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:07.837396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d3719a63-3b82-4cbb-8d2f-71c743df1557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:07.895922 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:07.918299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1105/2223] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:07.923672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab35515f-4e7d-42ce-9767-88646b87794b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:07.978046 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:08.017449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab35515f-4e7d-42ce-9767-88646b87794b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.025426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b831dac2-a995-41c6-82b3-545aa333511e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.026503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b831dac2-a995-41c6-82b3-545aa333511e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.027049 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b831dac2-a995-41c6-82b3-545aa333511e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.027745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b831dac2-a995-41c6-82b3-545aa333511e admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.028953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b831dac2-a995-41c6-82b3-545aa333511e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.032694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b831dac2-a995-41c6-82b3-545aa333511e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.081346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1119/2224] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:07 2022] POST /identity/v3/projects => generated 387 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:08.086192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3719a63-3b82-4cbb-8d2f-71c743df1557 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.099065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3719a63-3b82-4cbb-8d2f-71c743df1557 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.170080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a01c76f-e9ca-44ad-87d5-6a9f3a124e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.170623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a01c76f-e9ca-44ad-87d5-6a9f3a124e43 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.170920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a01c76f-e9ca-44ad-87d5-6a9f3a124e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.171192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a01c76f-e9ca-44ad-87d5-6a9f3a124e43 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.171866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a01c76f-e9ca-44ad-87d5-6a9f3a124e43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.173876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a01c76f-e9ca-44ad-87d5-6a9f3a124e43 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.206345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1106/2225] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:08 2022] POST /identity/v3/users => generated 294 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:08.211120 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b831dac2-a995-41c6-82b3-545aa333511e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.224078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b831dac2-a995-41c6-82b3-545aa333511e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.229117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d235b6-cf75-42f0-95b6-af95d2d7399d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.229750 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d235b6-cf75-42f0-95b6-af95d2d7399d admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.230035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d235b6-cf75-42f0-95b6-af95d2d7399d admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.230314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d235b6-cf75-42f0-95b6-af95d2d7399d admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.231010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d235b6-cf75-42f0-95b6-af95d2d7399d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.232668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d235b6-cf75-42f0-95b6-af95d2d7399d admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.247179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1120/2226] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:08.253872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a01c76f-e9ca-44ad-87d5-6a9f3a124e43 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.266932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a01c76f-e9ca-44ad-87d5-6a9f3a124e43 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.272851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cdb143-3664-4622-9db7-3dedc8a13cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.273550 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdb143-3664-4622-9db7-3dedc8a13cea admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.273990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdb143-3664-4622-9db7-3dedc8a13cea admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.274420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdb143-3664-4622-9db7-3dedc8a13cea admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.275377 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cdb143-3664-4622-9db7-3dedc8a13cea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.277920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cdb143-3664-4622-9db7-3dedc8a13cea admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.292485 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1107/2227] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:08.300297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d235b6-cf75-42f0-95b6-af95d2d7399d admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.318366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d235b6-cf75-42f0-95b6-af95d2d7399d admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.326609 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f356e5-b00d-4cce-b6f1-6978560f97ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.327799 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f356e5-b00d-4cce-b6f1-6978560f97ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.328328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f356e5-b00d-4cce-b6f1-6978560f97ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.328827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f356e5-b00d-4cce-b6f1-6978560f97ab admin admin] PATH_INFO: `/v3/projects/aae43589926f4aaab73e3db27741c5e4/users/31a24e276986442c92b58acef114a341/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.356601 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f356e5-b00d-4cce-b6f1-6978560f97ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=aae43589926f4aaab73e3db27741c5e4, user_id=31a24e276986442c92b58acef114a341, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.361643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f356e5-b00d-4cce-b6f1-6978560f97ab admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.385411 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1121/2228] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:08 2022] PUT /identity/v3/projects/aae43589926f4aaab73e3db27741c5e4/users/31a24e276986442c92b58acef114a341/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:08.398514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2d5da-f4f4-4841-8cc1-ae1e608c53c3 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.399178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2d5da-f4f4-4841-8cc1-ae1e608c53c3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.399178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2d5da-f4f4-4841-8cc1-ae1e608c53c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.436048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-71e2d5da-f4f4-4841-8cc1-ae1e608c53c3 None None] MFA Rules not processed for user `31a24e276986442c92b58acef114a341`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:08.437342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-71e2d5da-f4f4-4841-8cc1-ae1e608c53c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:08.516890 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1108/2229] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:08.522614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f356e5-b00d-4cce-b6f1-6978560f97ab admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.597474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f356e5-b00d-4cce-b6f1-6978560f97ab admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.602677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d6404-c2b1-4e9a-a688-06f8fe5f21ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.603376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d6404-c2b1-4e9a-a688-06f8fe5f21ca admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.603693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d6404-c2b1-4e9a-a688-06f8fe5f21ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.604006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d6404-c2b1-4e9a-a688-06f8fe5f21ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.604712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62d6404-c2b1-4e9a-a688-06f8fe5f21ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.606779 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62d6404-c2b1-4e9a-a688-06f8fe5f21ca admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.640401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1122/2230] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:08 2022] POST /identity/v3/projects => generated 387 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:08.644867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e2d5da-f4f4-4841-8cc1-ae1e608c53c3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.657464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e2d5da-f4f4-4841-8cc1-ae1e608c53c3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.718625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:08.727178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eab3997-fd80-46dd-8049-7507989be156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.728074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eab3997-fd80-46dd-8049-7507989be156 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.728541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eab3997-fd80-46dd-8049-7507989be156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.729030 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eab3997-fd80-46dd-8049-7507989be156 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.729754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eab3997-fd80-46dd-8049-7507989be156 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.732096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eab3997-fd80-46dd-8049-7507989be156 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.764296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1109/2231] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:08 2022] POST /identity/v3/users => generated 300 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:08.765634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d6404-c2b1-4e9a-a688-06f8fe5f21ca admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.780379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d6404-c2b1-4e9a-a688-06f8fe5f21ca admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.785971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f020c8-1fed-4454-88f5-333423637b72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.786820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f020c8-1fed-4454-88f5-333423637b72 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.787297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f020c8-1fed-4454-88f5-333423637b72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.787727 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f020c8-1fed-4454-88f5-333423637b72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.788585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f020c8-1fed-4454-88f5-333423637b72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.790658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f020c8-1fed-4454-88f5-333423637b72 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.808616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1123/2232] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:08.811195 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eab3997-fd80-46dd-8049-7507989be156 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.825012 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eab3997-fd80-46dd-8049-7507989be156 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.831477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974cbd56-a0ca-423d-a545-f05eec1e6d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:08.832596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974cbd56-a0ca-423d-a545-f05eec1e6d25 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:08.833048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974cbd56-a0ca-423d-a545-f05eec1e6d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:08.833514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974cbd56-a0ca-423d-a545-f05eec1e6d25 admin admin] PATH_INFO: `/v3/projects/92470fb5fa5c436db0958cb7510ea33e/users/0640e184268b4fc69f7590a1f97649e5/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:08.863538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974cbd56-a0ca-423d-a545-f05eec1e6d25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92470fb5fa5c436db0958cb7510ea33e, user_id=0640e184268b4fc69f7590a1f97649e5, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:08.867139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974cbd56-a0ca-423d-a545-f05eec1e6d25 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:08.890385 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1110/2233] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:08 2022] PUT /identity/v3/projects/92470fb5fa5c436db0958cb7510ea33e/users/0640e184268b4fc69f7590a1f97649e5/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:08.896002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f020c8-1fed-4454-88f5-333423637b72 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:08.911520 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f020c8-1fed-4454-88f5-333423637b72 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:08.985960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:09.001707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d17b459-39ec-47f7-aa4d-b1462a44f5c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:09.003506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17b459-39ec-47f7-aa4d-b1462a44f5c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:09.004399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17b459-39ec-47f7-aa4d-b1462a44f5c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:09.004852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17b459-39ec-47f7-aa4d-b1462a44f5c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:09.005688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d17b459-39ec-47f7-aa4d-b1462a44f5c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:09.008685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d17b459-39ec-47f7-aa4d-b1462a44f5c9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:09.023041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1124/2234] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:09.028008 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974cbd56-a0ca-423d-a545-f05eec1e6d25 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:09.046619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974cbd56-a0ca-423d-a545-f05eec1e6d25 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:09.055505 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:09.056340 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:09.056719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:09.057141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] PATH_INFO: `/v3/projects/92470fb5fa5c436db0958cb7510ea33e/users/0640e184268b4fc69f7590a1f97649e5/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:09.059731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92470fb5fa5c436db0958cb7510ea33e, user_id=0640e184268b4fc69f7590a1f97649e5, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:09.063290 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:09.081520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1111/2235] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:09 2022] PUT /identity/v3/projects/92470fb5fa5c436db0958cb7510ea33e/users/0640e184268b4fc69f7590a1f97649e5/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:09.099957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3413defc-bd67-484e-9e79-7139b984256d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:09.100449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3413defc-bd67-484e-9e79-7139b984256d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:09.100885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3413defc-bd67-484e-9e79-7139b984256d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:09.142819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-3413defc-bd67-484e-9e79-7139b984256d None None] MFA Rules not processed for user `0640e184268b4fc69f7590a1f97649e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:09.144856 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3413defc-bd67-484e-9e79-7139b984256d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:09.262093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1125/2236] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:09 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:09.275717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:09.277999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:09.365915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b061917-283d-4071-a102-f78cc283fec0 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:09.371335 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:09.372085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:09.372430 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:09.372760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:09.384265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:09.385760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:09.399932 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1112/2237] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:09 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:10.098905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3413defc-bd67-484e-9e79-7139b984256d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:10.100633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3413defc-bd67-484e-9e79-7139b984256d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:10.114728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3413defc-bd67-484e-9e79-7139b984256d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:10.118657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130da8cd-a4b1-44d3-806f-14913b1f9a96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:10.119337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130da8cd-a4b1-44d3-806f-14913b1f9a96 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:10.119612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130da8cd-a4b1-44d3-806f-14913b1f9a96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:10.119881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130da8cd-a4b1-44d3-806f-14913b1f9a96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:10.192941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130da8cd-a4b1-44d3-806f-14913b1f9a96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:10.194439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130da8cd-a4b1-44d3-806f-14913b1f9a96 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:10.207499 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1126/2238] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:10 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:11.932723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:11.935480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:12.042293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaabaa14-d62a-4b55-8d88-2bf14074799a service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:12.047282 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:12.047984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:12.048263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:12.048538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] PATH_INFO: `/v3/users/cb60e6e7ccb44422b60640c0b3ebdc9a` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:12.051071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb60e6e7ccb44422b60640c0b3ebdc9a)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:12.053062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:12.113745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb60e6e7ccb44422b60640c0b3ebdc9a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.121886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:12.125032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb60e6e7ccb44422b60640c0b3ebdc9a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.125918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb60e6e7ccb44422b60640c0b3ebdc9a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.127013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb60e6e7ccb44422b60640c0b3ebdc9a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.137066 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb60e6e7ccb44422b60640c0b3ebdc9a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.160150 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1113/2239] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:11 2022] DELETE /identity/v3/users/cb60e6e7ccb44422b60640c0b3ebdc9a => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:12.165297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8dda68-e4c1-4a37-8468-daa11151bb3a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:12.165957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8dda68-e4c1-4a37-8468-daa11151bb3a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:12.165957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8dda68-e4c1-4a37-8468-daa11151bb3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:12.186881 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8c8dda68-e4c1-4a37-8468-daa11151bb3a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:12.188213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8c8dda68-e4c1-4a37-8468-daa11151bb3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:12.265518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1127/2240] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:12.280439 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:12.282689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:12.371559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7181fd27-36d4-46e2-b6b2-81a0288d3c6f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:12.375379 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ff0a7-bf16-4321-a47d-9e0622d04100 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:12.376022 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ff0a7-bf16-4321-a47d-9e0622d04100 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:12.376304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ff0a7-bf16-4321-a47d-9e0622d04100 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:12.376579 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ff0a7-bf16-4321-a47d-9e0622d04100 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:12.389284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068ff0a7-bf16-4321-a47d-9e0622d04100 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:12.392302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068ff0a7-bf16-4321-a47d-9e0622d04100 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:12.413173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1114/2241] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:12.791059 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8dda68-e4c1-4a37-8468-daa11151bb3a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:12.793331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8c8dda68-e4c1-4a37-8468-daa11151bb3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:12.813206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8dda68-e4c1-4a37-8468-daa11151bb3a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:12.822583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:12.823747 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:12.824209 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:12.824681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] PATH_INFO: `/v3/projects/72832e3d4e51418bae334ade727d8446` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:12.827409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] RBAC: Authorizing `identity:delete_project(project_id=72832e3d4e51418bae334ade727d8446)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:12.831371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:12.849472 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:12.895129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72832e3d4e51418bae334ade727d8446', 'request_id': 'req-e07ca032-dbd1-4a79-bd0c-8a165243b54f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.901760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72832e3d4e51418bae334ade727d8446', 'request_id': 'req-e07ca032-dbd1-4a79-bd0c-8a165243b54f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.911226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72832e3d4e51418bae334ade727d8446', 'request_id': 'req-e07ca032-dbd1-4a79-bd0c-8a165243b54f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.926158 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] The token cache is being invalidate because project 72832e3d4e51418bae334ade727d8446 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:12.926414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.927111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:12.928682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1128/2242] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:12 2022] DELETE /identity/v3/projects/72832e3d4e51418bae334ade727d8446 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:12.933599 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ff0a7-bf16-4321-a47d-9e0622d04100 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:12.951390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ff0a7-bf16-4321-a47d-9e0622d04100 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:13.053250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:13.054331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:13.054837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:13.055161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] PATH_INFO: `/v3/users/4b79464667834bc290bbd51298654961` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:13.057097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b79464667834bc290bbd51298654961)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:13.059298 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:13.133963 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b79464667834bc290bbd51298654961'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.144068 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b79464667834bc290bbd51298654961'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.144950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b79464667834bc290bbd51298654961'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.145574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b79464667834bc290bbd51298654961'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.156631 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b1278296-e8d4-4d27-b1f2-b47d7c49d09a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b79464667834bc290bbd51298654961'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.177348 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:13.193049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1115/2243] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:12 2022] DELETE /identity/v3/users/4b79464667834bc290bbd51298654961 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:13.651483 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:13.653434 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:13.670591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07ca032-dbd1-4a79-bd0c-8a165243b54f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:13.743576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:13.744243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:13.744529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:13.744805 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] PATH_INFO: `/v3/projects/ef53e1b42c3840b7aaae6aea6c6fe274` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:13.746191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef53e1b42c3840b7aaae6aea6c6fe274)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:13.748479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:13.804854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef53e1b42c3840b7aaae6aea6c6fe274', 'request_id': 'req-a8392d09-4e27-4323-b9a5-bb08619e64ff'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.812647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef53e1b42c3840b7aaae6aea6c6fe274', 'request_id': 'req-a8392d09-4e27-4323-b9a5-bb08619e64ff'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.823581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef53e1b42c3840b7aaae6aea6c6fe274', 'request_id': 'req-a8392d09-4e27-4323-b9a5-bb08619e64ff'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.838676 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] The token cache is being invalidate because project ef53e1b42c3840b7aaae6aea6c6fe274 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:13.838962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.839687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8392d09-4e27-4323-b9a5-bb08619e64ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:13.841637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1129/2244] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:13 2022] DELETE /identity/v3/projects/ef53e1b42c3840b7aaae6aea6c6fe274 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:13.849437 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a33ad-a1e7-4a05-a27e-3c0884a7c55b None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:13.849785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a33ad-a1e7-4a05-a27e-3c0884a7c55b None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:13.850122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a33ad-a1e7-4a05-a27e-3c0884a7c55b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:13.873450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a51a33ad-a1e7-4a05-a27e-3c0884a7c55b None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:13.875131 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a51a33ad-a1e7-4a05-a27e-3c0884a7c55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:13.970380 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1116/2245] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:13.984800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e42a3c7-b048-4767-85fd-7f0a6e7614f7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:13.985297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e42a3c7-b048-4767-85fd-7f0a6e7614f7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:13.985792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e42a3c7-b048-4767-85fd-7f0a6e7614f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.021007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9e42a3c7-b048-4767-85fd-7f0a6e7614f7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:14.023240 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9e42a3c7-b048-4767-85fd-7f0a6e7614f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:14.039193 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1130/2246] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:14.043853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51a33ad-a1e7-4a05-a27e-3c0884a7c55b None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.055709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51a33ad-a1e7-4a05-a27e-3c0884a7c55b None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.061004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.061642 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.061923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.062223 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.062929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:14.063233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.065802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.081354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1117/2247] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:48:14.086949 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e42a3c7-b048-4767-85fd-7f0a6e7614f7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.098372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e42a3c7-b048-4767-85fd-7f0a6e7614f7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.103770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a9fb10-454e-4c06-b98d-d5a7315de45a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.104414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a9fb10-454e-4c06-b98d-d5a7315de45a admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.104691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a9fb10-454e-4c06-b98d-d5a7315de45a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.104966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a9fb10-454e-4c06-b98d-d5a7315de45a admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.105625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a9fb10-454e-4c06-b98d-d5a7315de45a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.107648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a9fb10-454e-4c06-b98d-d5a7315de45a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.136121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:14.149963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1131/2248] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:14 2022] POST /identity/v3/projects => generated 373 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:14.151371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.163743 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30eb5e7a-aa9e-4a41-bfbf-8e65f84d8846 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.240280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6deabd5-beb2-4b11-8f9e-b5eff5fbb166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.240898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6deabd5-beb2-4b11-8f9e-b5eff5fbb166 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.241219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6deabd5-beb2-4b11-8f9e-b5eff5fbb166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.241593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6deabd5-beb2-4b11-8f9e-b5eff5fbb166 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.242639 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6deabd5-beb2-4b11-8f9e-b5eff5fbb166 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.244708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6deabd5-beb2-4b11-8f9e-b5eff5fbb166 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.274637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1118/2249] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:14 2022] POST /identity/v3/users => generated 287 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:14.281213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a9fb10-454e-4c06-b98d-d5a7315de45a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.294523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a9fb10-454e-4c06-b98d-d5a7315de45a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.300393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639d7a5b-9c44-4835-aec8-5a71fabc6411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.301086 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639d7a5b-9c44-4835-aec8-5a71fabc6411 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.301417 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639d7a5b-9c44-4835-aec8-5a71fabc6411 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.301742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639d7a5b-9c44-4835-aec8-5a71fabc6411 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.302500 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639d7a5b-9c44-4835-aec8-5a71fabc6411 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.304358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639d7a5b-9c44-4835-aec8-5a71fabc6411 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.318730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1132/2250] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:14.323248 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6deabd5-beb2-4b11-8f9e-b5eff5fbb166 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.335650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6deabd5-beb2-4b11-8f9e-b5eff5fbb166 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.340997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d34057-13d1-43c9-99ec-e8b3364e48af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.341634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d34057-13d1-43c9-99ec-e8b3364e48af admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.341934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d34057-13d1-43c9-99ec-e8b3364e48af admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.342183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d34057-13d1-43c9-99ec-e8b3364e48af admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.342904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d34057-13d1-43c9-99ec-e8b3364e48af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.344804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d34057-13d1-43c9-99ec-e8b3364e48af admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.358161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1119/2251] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:14.363541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639d7a5b-9c44-4835-aec8-5a71fabc6411 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.377802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639d7a5b-9c44-4835-aec8-5a71fabc6411 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.383082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90178852-409b-4066-9343-15b676a07441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.383768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90178852-409b-4066-9343-15b676a07441 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.384050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90178852-409b-4066-9343-15b676a07441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.384329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90178852-409b-4066-9343-15b676a07441 admin admin] PATH_INFO: `/v3/projects/a24c5dede7744e19a592f0c12adf7fd3/users/1b552f61630f4d41abf53beecfb1448c/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.402002 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90178852-409b-4066-9343-15b676a07441 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a24c5dede7744e19a592f0c12adf7fd3, user_id=1b552f61630f4d41abf53beecfb1448c, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.404765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90178852-409b-4066-9343-15b676a07441 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.421384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1133/2252] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:14 2022] PUT /identity/v3/projects/a24c5dede7744e19a592f0c12adf7fd3/users/1b552f61630f4d41abf53beecfb1448c/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:14.433610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8219e2a5-1948-4ccc-a569-9b41926c4359 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.433902 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8219e2a5-1948-4ccc-a569-9b41926c4359 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.434157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8219e2a5-1948-4ccc-a569-9b41926c4359 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.472961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8219e2a5-1948-4ccc-a569-9b41926c4359 None None] MFA Rules not processed for user `1b552f61630f4d41abf53beecfb1448c`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:14.475074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8219e2a5-1948-4ccc-a569-9b41926c4359 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:14.566060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:14.597745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1120/2253] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:14.603085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90178852-409b-4066-9343-15b676a07441 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.710916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90178852-409b-4066-9343-15b676a07441 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.716430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.717029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.717343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.717590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.718225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.720591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.755282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1134/2254] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:14 2022] POST /identity/v3/projects => generated 375 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:14.760851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8219e2a5-1948-4ccc-a569-9b41926c4359 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.774625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8219e2a5-1948-4ccc-a569-9b41926c4359 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.776368 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.779327 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:14.848577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2206834b-e221-4421-bbd1-b96465907474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.849386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2206834b-e221-4421-bbd1-b96465907474 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.849839 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2206834b-e221-4421-bbd1-b96465907474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.850236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2206834b-e221-4421-bbd1-b96465907474 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.851181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2206834b-e221-4421-bbd1-b96465907474 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.853544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2206834b-e221-4421-bbd1-b96465907474 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.885912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1121/2255] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:14 2022] POST /identity/v3/users => generated 294 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:14.886100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fab3b6-4d51-440e-b83a-f0be45845f6a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.887220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2206834b-e221-4421-bbd1-b96465907474 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.890280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.891835 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.892325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.894047 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.897961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:14.899350 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2206834b-e221-4421-bbd1-b96465907474 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.904283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf35f6f-1bb7-4b4c-bb95-c91b73b178bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.904911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf35f6f-1bb7-4b4c-bb95-c91b73b178bd admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.905189 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf35f6f-1bb7-4b4c-bb95-c91b73b178bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.905463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf35f6f-1bb7-4b4c-bb95-c91b73b178bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.906126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf35f6f-1bb7-4b4c-bb95-c91b73b178bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.907845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf35f6f-1bb7-4b4c-bb95-c91b73b178bd admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.920320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1122/2256] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:14.925108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf35f6f-1bb7-4b4c-bb95-c91b73b178bd admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.936009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf35f6f-1bb7-4b4c-bb95-c91b73b178bd admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:14.941797 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639a5fd2-a721-472d-9e65-31b59c616e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:14.942512 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a5fd2-a721-472d-9e65-31b59c616e6c admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:14.942882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a5fd2-a721-472d-9e65-31b59c616e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:14.943116 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a5fd2-a721-472d-9e65-31b59c616e6c admin admin] PATH_INFO: `/v3/projects/d7894ac199604e5ea423ae2b1bc0f501/users/ec2ed7e40a0d4353875174acc019eaf6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:14.961992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639a5fd2-a721-472d-9e65-31b59c616e6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7894ac199604e5ea423ae2b1bc0f501, user_id=ec2ed7e40a0d4353875174acc019eaf6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.964796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639a5fd2-a721-472d-9e65-31b59c616e6c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:14.981776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1123/2257] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:14 2022] PUT /identity/v3/projects/d7894ac199604e5ea423ae2b1bc0f501/users/ec2ed7e40a0d4353875174acc019eaf6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:14.986694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639a5fd2-a721-472d-9e65-31b59c616e6c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:14.998018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:14.999820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:15.014986 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1135/2258] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:14 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:15.083654 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639a5fd2-a721-472d-9e65-31b59c616e6c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:15.091870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c8336f-a737-4d8f-b50a-3aa015f99860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:15.092844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c8336f-a737-4d8f-b50a-3aa015f99860 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:15.093297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c8336f-a737-4d8f-b50a-3aa015f99860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:15.093745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c8336f-a737-4d8f-b50a-3aa015f99860 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:15.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c8336f-a737-4d8f-b50a-3aa015f99860 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:15.098663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c8336f-a737-4d8f-b50a-3aa015f99860 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:15.117265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1124/2259] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:15.122096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:15.130340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:15.136507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7cc1b6-e589-4e72-a96a-92d0d5cc97e6 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:15.145076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:15.146246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:15.147014 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:15.147469 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] PATH_INFO: `/v3/projects/d7894ac199604e5ea423ae2b1bc0f501/users/ec2ed7e40a0d4353875174acc019eaf6/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:15.151723 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7894ac199604e5ea423ae2b1bc0f501, user_id=ec2ed7e40a0d4353875174acc019eaf6, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:15.156353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:15.175697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1136/2260] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:15 2022] PUT /identity/v3/projects/d7894ac199604e5ea423ae2b1bc0f501/users/ec2ed7e40a0d4353875174acc019eaf6/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:15.191618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39b1e9-3811-458f-8974-ee5faa10bdbc None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:15.192112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39b1e9-3811-458f-8974-ee5faa10bdbc None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:15.192526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39b1e9-3811-458f-8974-ee5faa10bdbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:15.244055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-aa39b1e9-3811-458f-8974-ee5faa10bdbc None None] MFA Rules not processed for user `ec2ed7e40a0d4353875174acc019eaf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:15.246267 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-aa39b1e9-3811-458f-8974-ee5faa10bdbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:15.302209 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:15.385777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1125/2261] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:15 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:15.407220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:15.410714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:15.509621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231ba44e-475d-4051-8f0d-99fa952d46ce admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:15.513660 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d070384-3790-4231-90ed-23ffc5372851 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:15.514305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d070384-3790-4231-90ed-23ffc5372851 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:15.514634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d070384-3790-4231-90ed-23ffc5372851 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:15.514906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d070384-3790-4231-90ed-23ffc5372851 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:15.596481 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d070384-3790-4231-90ed-23ffc5372851 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:15.599009 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d070384-3790-4231-90ed-23ffc5372851 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:15.612880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1137/2262] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:17.699933 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa39b1e9-3811-458f-8974-ee5faa10bdbc None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:17.701543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-aa39b1e9-3811-458f-8974-ee5faa10bdbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:17.790156 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa39b1e9-3811-458f-8974-ee5faa10bdbc None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:17.794991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:17.795648 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:17.795924 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:17.796193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] PATH_INFO: `/v3/users/de163c501d594e6fb488e72f3876a82f` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:17.797455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de163c501d594e6fb488e72f3876a82f)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:17.799532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:17.868339 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de163c501d594e6fb488e72f3876a82f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:17.878473 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de163c501d594e6fb488e72f3876a82f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:17.879403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de163c501d594e6fb488e72f3876a82f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:17.880140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de163c501d594e6fb488e72f3876a82f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:17.890320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de163c501d594e6fb488e72f3876a82f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:17.912191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1126/2263] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:17 2022] DELETE /identity/v3/users/de163c501d594e6fb488e72f3876a82f => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:17.917118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca9075e-1717-4812-bd95-312796f3de3f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:17.917686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca9075e-1717-4812-bd95-312796f3de3f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:17.917686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca9075e-1717-4812-bd95-312796f3de3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:17.939386 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1ca9075e-1717-4812-bd95-312796f3de3f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:17.941314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1ca9075e-1717-4812-bd95-312796f3de3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:18.010774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1138/2264] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:18.022804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:18.024417 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:18.082151 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:18.097717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdf2a57-70fa-48a7-9915-4b991975fd77 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:18.104586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:18.105293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:18.105617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:18.105934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:18.117665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:18.119541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:18.131310 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1127/2265] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:18.471102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca9075e-1717-4812-bd95-312796f3de3f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:18.473289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1ca9075e-1717-4812-bd95-312796f3de3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:18.491178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca9075e-1717-4812-bd95-312796f3de3f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:18.497722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:18.498839 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:18.499329 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:18.499809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] PATH_INFO: `/v3/projects/d681ca329da2403f9710bb2cde758948` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:18.501521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d681ca329da2403f9710bb2cde758948)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:18.504784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:18.531095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:18.532833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:18.556555 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a2b973-9d6f-4c7a-8478-37b258c54ea8 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:18.565677 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:18.566734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:18.567140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:18.567514 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] PATH_INFO: `/v3/users/1b552f61630f4d41abf53beecfb1448c` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:18.569334 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b552f61630f4d41abf53beecfb1448c)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:18.572769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:18.578296 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:18.591318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd681ca329da2403f9710bb2cde758948', 'request_id': 'req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.602514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd681ca329da2403f9710bb2cde758948', 'request_id': 'req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.615358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd681ca329da2403f9710bb2cde758948', 'request_id': 'req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.636929 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] The token cache is being invalidate because project d681ca329da2403f9710bb2cde758948 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:18.637319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.638389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.640886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1139/2266] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:18 2022] DELETE /identity/v3/projects/d681ca329da2403f9710bb2cde758948 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:18.646836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:18.649006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:18.652833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b552f61630f4d41abf53beecfb1448c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.665438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b552f61630f4d41abf53beecfb1448c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.666544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b552f61630f4d41abf53beecfb1448c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.667329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b552f61630f4d41abf53beecfb1448c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.678744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4c17c25b-330a-4ac0-b034-54d0cf937242 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b552f61630f4d41abf53beecfb1448c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.708792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1128/2267] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:18 2022] DELETE /identity/v3/users/1b552f61630f4d41abf53beecfb1448c => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:18.711683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e27d9-36ad-4efe-ab54-2e71815adc70 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:18.712112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e27d9-36ad-4efe-ab54-2e71815adc70 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:18.712527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e27d9-36ad-4efe-ab54-2e71815adc70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:18.737681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-f38e27d9-36ad-4efe-ab54-2e71815adc70 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:18.739446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f38e27d9-36ad-4efe-ab54-2e71815adc70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:18.744669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909619b2-e11e-4bcb-bf4f-27bee0b17d4b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:18.820374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1129/2268] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:18.823514 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:18.824506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:18.824956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:18.825331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] PATH_INFO: `/v3/users/31a24e276986442c92b58acef114a341` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:18.826924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31a24e276986442c92b58acef114a341)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:18.829119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:18.832643 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38e27d9-36ad-4efe-ab54-2e71815adc70 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:18.834944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-f38e27d9-36ad-4efe-ab54-2e71815adc70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:18.905577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31a24e276986442c92b58acef114a341'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.916496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31a24e276986442c92b58acef114a341'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.917771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31a24e276986442c92b58acef114a341'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.919054 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31a24e276986442c92b58acef114a341'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.930927 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38e27d9-36ad-4efe-ab54-2e71815adc70 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:18.933840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31a24e276986442c92b58acef114a341'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:18.937055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:18.937911 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:18.938299 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:18.938795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:18.941729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:18.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:18.961629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:18.963945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:18.966041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1140/2269] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:18 2022] DELETE /identity/v3/users/31a24e276986442c92b58acef114a341 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:18.978530 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1130/2270] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:19.370921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:19.372741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:19.481767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:19.484095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:19.497653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:19.501179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fb715d-07cf-4f4e-926c-61018622b2d8 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:19.508065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb51b196-96cf-446d-b4e8-133baa4292ea service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:19.511093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:19.512256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:19.512688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:19.513124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] PATH_INFO: `/v3/projects/aae43589926f4aaab73e3db27741c5e4` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:19.515226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aae43589926f4aaab73e3db27741c5e4)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:19.519505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:19.523129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:19.524449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:19.524854 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:19.525399 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] PATH_INFO: `/v3/projects/a24c5dede7744e19a592f0c12adf7fd3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:19.529125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a24c5dede7744e19a592f0c12adf7fd3)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:19.532728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:19.604140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aae43589926f4aaab73e3db27741c5e4', 'request_id': 'req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.609872 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a24c5dede7744e19a592f0c12adf7fd3', 'request_id': 'req-096bd689-418b-4ccb-943c-1c01439e20e9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.615532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aae43589926f4aaab73e3db27741c5e4', 'request_id': 'req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.624587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a24c5dede7744e19a592f0c12adf7fd3', 'request_id': 'req-096bd689-418b-4ccb-943c-1c01439e20e9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.625610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aae43589926f4aaab73e3db27741c5e4', 'request_id': 'req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.635887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a24c5dede7744e19a592f0c12adf7fd3', 'request_id': 'req-096bd689-418b-4ccb-943c-1c01439e20e9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.644558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] The token cache is being invalidate because project aae43589926f4aaab73e3db27741c5e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:19.645098 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.646106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.648721 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1141/2271] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:19 2022] DELETE /identity/v3/projects/aae43589926f4aaab73e3db27741c5e4 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:19.653656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:19.654763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] The token cache is being invalidate because project a24c5dede7744e19a592f0c12adf7fd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:19.655140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.655867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:19.656203 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.658777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1131/2272] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:19 2022] DELETE /identity/v3/projects/a24c5dede7744e19a592f0c12adf7fd3 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:19.664429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:19.666764 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:19.745271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eefd3c3-fafd-4eb6-9a19-601750f1f1e2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:19.772826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096bd689-418b-4ccb-943c-1c01439e20e9 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:19.778220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:19.778879 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:19.779250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:19.779408 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] PATH_INFO: `/v3/users/ec2ed7e40a0d4353875174acc019eaf6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:19.780683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec2ed7e40a0d4353875174acc019eaf6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:19.782704 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:19.791592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:19.843745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec2ed7e40a0d4353875174acc019eaf6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.849427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:19.850443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:19.850942 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:19.851394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] PATH_INFO: `/v3/users/0640e184268b4fc69f7590a1f97649e5` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:19.853800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=0640e184268b4fc69f7590a1f97649e5)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:19.856716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:19.856970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec2ed7e40a0d4353875174acc019eaf6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.857815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec2ed7e40a0d4353875174acc019eaf6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.858760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec2ed7e40a0d4353875174acc019eaf6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.869057 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec2ed7e40a0d4353875174acc019eaf6'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.898666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1132/2273] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:19 2022] DELETE /identity/v3/users/ec2ed7e40a0d4353875174acc019eaf6 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:19.937794 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0640e184268b4fc69f7590a1f97649e5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.951755 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0640e184268b4fc69f7590a1f97649e5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.952998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0640e184268b4fc69f7590a1f97649e5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.954062 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0640e184268b4fc69f7590a1f97649e5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.966431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0640e184268b4fc69f7590a1f97649e5'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:19.997051 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1142/2274] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:19 2022] DELETE /identity/v3/users/0640e184268b4fc69f7590a1f97649e5 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:20.370172 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:20.372712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:20.384692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:20.459128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:20.461593 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:20.521227 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7819e16f-54c8-4c61-87e5-7273295e7607 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:20.587722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d135f874-e87c-4c92-893e-326919fdd0dc admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:20.596578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:20.597786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.598226 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:20.598432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.598786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] PATH_INFO: `/v3/projects/92470fb5fa5c436db0958cb7510ea33e` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.598948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.599183 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.599464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] PATH_INFO: `/v3/projects/d7894ac199604e5ea423ae2b1bc0f501` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.600688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7894ac199604e5ea423ae2b1bc0f501)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:20.600869 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=92470fb5fa5c436db0958cb7510ea33e)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:20.602719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:20.604446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:20.673612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7894ac199604e5ea423ae2b1bc0f501', 'request_id': 'req-6d22b89e-0b58-4c65-a952-027a5262c24b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.683085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7894ac199604e5ea423ae2b1bc0f501', 'request_id': 'req-6d22b89e-0b58-4c65-a952-027a5262c24b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.685104 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92470fb5fa5c436db0958cb7510ea33e', 'request_id': 'req-12ef755d-0639-4322-8a13-f1442241afe7'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.693237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:20.693585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7894ac199604e5ea423ae2b1bc0f501', 'request_id': 'req-6d22b89e-0b58-4c65-a952-027a5262c24b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.696967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92470fb5fa5c436db0958cb7510ea33e', 'request_id': 'req-12ef755d-0639-4322-8a13-f1442241afe7'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.711129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92470fb5fa5c436db0958cb7510ea33e', 'request_id': 'req-12ef755d-0639-4322-8a13-f1442241afe7'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.715048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] The token cache is being invalidate because project d7894ac199604e5ea423ae2b1bc0f501 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:20.715404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.716444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6d22b89e-0b58-4c65-a952-027a5262c24b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.718863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1133/2275] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:20 2022] DELETE /identity/v3/projects/d7894ac199604e5ea423ae2b1bc0f501 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:20.726653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f468cab-eb7a-4573-a44f-b27d164d10c2 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.727050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f468cab-eb7a-4573-a44f-b27d164d10c2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.727504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f468cab-eb7a-4573-a44f-b27d164d10c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.733027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] The token cache is being invalidate because project 92470fb5fa5c436db0958cb7510ea33e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:20.733507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.734598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-12ef755d-0639-4322-8a13-f1442241afe7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:20.737557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1143/2276] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:20 2022] DELETE /identity/v3/projects/92470fb5fa5c436db0958cb7510ea33e => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:20.746738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a19e512-cc30-4e71-aa5c-2f7b4a70c930 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.747190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a19e512-cc30-4e71-aa5c-2f7b4a70c930 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.747641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a19e512-cc30-4e71-aa5c-2f7b4a70c930 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.760199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8f468cab-eb7a-4573-a44f-b27d164d10c2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:20.761516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8f468cab-eb7a-4573-a44f-b27d164d10c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:20.779364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-1a19e512-cc30-4e71-aa5c-2f7b4a70c930 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:20.781294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-1a19e512-cc30-4e71-aa5c-2f7b4a70c930 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:20.826636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:20.859936 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1134/2277] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:20.869500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2f2b5-2e1a-4e2e-97c5-6353f9980c14 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.869877 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2f2b5-2e1a-4e2e-97c5-6353f9980c14 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.870222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2f2b5-2e1a-4e2e-97c5-6353f9980c14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.875530 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1144/2278] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:20.889297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba56468-d7e6-4a3e-9e36-f7a9200700b7 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.889720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba56468-d7e6-4a3e-9e36-f7a9200700b7 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.890119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba56468-d7e6-4a3e-9e36-f7a9200700b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.898157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-70e2f2b5-2e1a-4e2e-97c5-6353f9980c14 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:20.899799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-70e2f2b5-2e1a-4e2e-97c5-6353f9980c14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:20.910247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1135/2279] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:20.915259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e2f2b5-2e1a-4e2e-97c5-6353f9980c14 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:20.916110 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-eba56468-d7e6-4a3e-9e36-f7a9200700b7 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:20.917455 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-eba56468-d7e6-4a3e-9e36-f7a9200700b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:20.930391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e2f2b5-2e1a-4e2e-97c5-6353f9980c14 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:20.936987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1145/2280] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:20.938376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba56468-d7e6-4a3e-9e36-f7a9200700b7 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:20.939997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:20.939997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.940337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.940578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.941383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:20.941790 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:20.943596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:20.956179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba56468-d7e6-4a3e-9e36-f7a9200700b7 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:20.963939 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1136/2281] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:48:20.964389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:20.965243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:20.965423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.965932 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.966417 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.967640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:20.968198 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:20.971720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:20.977624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570bce6a-2eaa-4f52-b60c-8a9f0664e38e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:20.982837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799b9e15-9169-4f2e-9bbd-8900082e3f04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:20.983459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b9e15-9169-4f2e-9bbd-8900082e3f04 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:20.983746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b9e15-9169-4f2e-9bbd-8900082e3f04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:20.984013 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799b9e15-9169-4f2e-9bbd-8900082e3f04 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:20.984686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799b9e15-9169-4f2e-9bbd-8900082e3f04 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:20.986904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799b9e15-9169-4f2e-9bbd-8900082e3f04 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:20.997745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1146/2282] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:48:20.999437 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.018227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3ea533-d473-4d2a-8ecc-d603e4a59ced admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.019804 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1137/2283] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:20 2022] POST /identity/v3/projects => generated 385 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:21.024471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799b9e15-9169-4f2e-9bbd-8900082e3f04 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.111629 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa21231-f654-494a-9e5b-7809e9c92e88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.112024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799b9e15-9169-4f2e-9bbd-8900082e3f04 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.112275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa21231-f654-494a-9e5b-7809e9c92e88 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.112560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa21231-f654-494a-9e5b-7809e9c92e88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.112906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa21231-f654-494a-9e5b-7809e9c92e88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.113485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa21231-f654-494a-9e5b-7809e9c92e88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.115504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa21231-f654-494a-9e5b-7809e9c92e88 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.117994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf0aa91-cc9c-4dde-bdc2-59eae1a89e70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.118765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf0aa91-cc9c-4dde-bdc2-59eae1a89e70 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.119314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf0aa91-cc9c-4dde-bdc2-59eae1a89e70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.119597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf0aa91-cc9c-4dde-bdc2-59eae1a89e70 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.120437 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf0aa91-cc9c-4dde-bdc2-59eae1a89e70 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.122945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf0aa91-cc9c-4dde-bdc2-59eae1a89e70 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.148958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1147/2284] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:20 2022] POST /identity/v3/projects => generated 367 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:21.151328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1138/2285] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:21 2022] POST /identity/v3/users => generated 293 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:21.154555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa21231-f654-494a-9e5b-7809e9c92e88 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.155541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf0aa91-cc9c-4dde-bdc2-59eae1a89e70 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.168135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf0aa91-cc9c-4dde-bdc2-59eae1a89e70 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.227163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:21.252453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa21231-f654-494a-9e5b-7809e9c92e88 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.256708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c96676-d33a-4bf2-888e-73f2aec1ce8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.257398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c96676-d33a-4bf2-888e-73f2aec1ce8e admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.257679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c96676-d33a-4bf2-888e-73f2aec1ce8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.258081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c96676-d33a-4bf2-888e-73f2aec1ce8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.258853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c96676-d33a-4bf2-888e-73f2aec1ce8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.260753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c96676-d33a-4bf2-888e-73f2aec1ce8e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.262225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3b2b76-d03f-4446-985c-5ba38b876ec5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.263686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3b2b76-d03f-4446-985c-5ba38b876ec5 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.264183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3b2b76-d03f-4446-985c-5ba38b876ec5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.264682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3b2b76-d03f-4446-985c-5ba38b876ec5 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.265832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3b2b76-d03f-4446-985c-5ba38b876ec5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.270337 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3b2b76-d03f-4446-985c-5ba38b876ec5 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.274200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1139/2286] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:21 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:21.278755 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c96676-d33a-4bf2-888e-73f2aec1ce8e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.292759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c96676-d33a-4bf2-888e-73f2aec1ce8e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.298663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9219d67-c2f3-4f53-8be9-57238ac768c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.299632 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9219d67-c2f3-4f53-8be9-57238ac768c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.299912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9219d67-c2f3-4f53-8be9-57238ac768c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.300184 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9219d67-c2f3-4f53-8be9-57238ac768c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.300845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9219d67-c2f3-4f53-8be9-57238ac768c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.302538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9219d67-c2f3-4f53-8be9-57238ac768c5 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.303405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1148/2287] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:21 2022] POST /identity/v3/users => generated 284 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:21.309131 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3b2b76-d03f-4446-985c-5ba38b876ec5 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.318431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1140/2288] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:21 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:21.319979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9219d67-c2f3-4f53-8be9-57238ac768c5 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.327774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3b2b76-d03f-4446-985c-5ba38b876ec5 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.334272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9219d67-c2f3-4f53-8be9-57238ac768c5 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.336864 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85df9dcd-1771-4dde-88bd-a09312df3a70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.337900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df9dcd-1771-4dde-88bd-a09312df3a70 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.338393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df9dcd-1771-4dde-88bd-a09312df3a70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.338943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df9dcd-1771-4dde-88bd-a09312df3a70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.340203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85df9dcd-1771-4dde-88bd-a09312df3a70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.340595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab37341-8493-43ca-9f9f-a6dad75a36a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.341431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab37341-8493-43ca-9f9f-a6dad75a36a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.341779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab37341-8493-43ca-9f9f-a6dad75a36a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.342115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab37341-8493-43ca-9f9f-a6dad75a36a9 admin admin] PATH_INFO: `/v3/projects/240ea8be5fe94613b0d9ca70005d49df/users/896c2d615013484da0a55a9dc8e4a65f/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.343476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85df9dcd-1771-4dde-88bd-a09312df3a70 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.357465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:21.361106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab37341-8493-43ca-9f9f-a6dad75a36a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=240ea8be5fe94613b0d9ca70005d49df, user_id=896c2d615013484da0a55a9dc8e4a65f, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.364566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab37341-8493-43ca-9f9f-a6dad75a36a9 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.366119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1149/2289] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:21 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:21.372567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85df9dcd-1771-4dde-88bd-a09312df3a70 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.391737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1141/2290] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:21 2022] PUT /identity/v3/projects/240ea8be5fe94613b0d9ca70005d49df/users/896c2d615013484da0a55a9dc8e4a65f/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:21.392026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85df9dcd-1771-4dde-88bd-a09312df3a70 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.398363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382d170-18df-4f54-ba8c-422e6044fa2a None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.398865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382d170-18df-4f54-ba8c-422e6044fa2a None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.399326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382d170-18df-4f54-ba8c-422e6044fa2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.435478 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2382d170-18df-4f54-ba8c-422e6044fa2a None None] MFA Rules not processed for user `896c2d615013484da0a55a9dc8e4a65f`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:21.437304 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2382d170-18df-4f54-ba8c-422e6044fa2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:21.476288 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36d8c82-fd1e-4adc-8c71-9572aa426984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.477311 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36d8c82-fd1e-4adc-8c71-9572aa426984 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.477814 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36d8c82-fd1e-4adc-8c71-9572aa426984 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.478249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36d8c82-fd1e-4adc-8c71-9572aa426984 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.479100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36d8c82-fd1e-4adc-8c71-9572aa426984 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.481103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36d8c82-fd1e-4adc-8c71-9572aa426984 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.498364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1150/2291] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:21 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:21.499753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36d8c82-fd1e-4adc-8c71-9572aa426984 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.512435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36d8c82-fd1e-4adc-8c71-9572aa426984 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.520153 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931f46c-6138-463b-a0fa-e51dd07e74e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.521556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931f46c-6138-463b-a0fa-e51dd07e74e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.521953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931f46c-6138-463b-a0fa-e51dd07e74e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.522455 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931f46c-6138-463b-a0fa-e51dd07e74e9 admin admin] PATH_INFO: `/v3/projects/e8a6e9bad74c4089b6c12b6e7d0cd1a6/users/f13e5850266448c095afbb3aa73b512b/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.524970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1142/2292] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:21.538077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382d170-18df-4f54-ba8c-422e6044fa2a None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.540120 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2382d170-18df-4f54-ba8c-422e6044fa2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:21.552229 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931f46c-6138-463b-a0fa-e51dd07e74e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8a6e9bad74c4089b6c12b6e7d0cd1a6, user_id=f13e5850266448c095afbb3aa73b512b, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.557251 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931f46c-6138-463b-a0fa-e51dd07e74e9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.580058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1151/2293] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:21 2022] PUT /identity/v3/projects/e8a6e9bad74c4089b6c12b6e7d0cd1a6/users/f13e5850266448c095afbb3aa73b512b/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:21.595398 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7e6cbc-3af3-44d4-9a68-d2cea5543220 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.596222 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7e6cbc-3af3-44d4-9a68-d2cea5543220 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.596460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7e6cbc-3af3-44d4-9a68-d2cea5543220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.616279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382d170-18df-4f54-ba8c-422e6044fa2a None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.633957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-bd7e6cbc-3af3-44d4-9a68-d2cea5543220 None None] MFA Rules not processed for user `f13e5850266448c095afbb3aa73b512b`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:21.635299 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-bd7e6cbc-3af3-44d4-9a68-d2cea5543220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:21.689929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9209ef-a2b2-4d25-b3bc-42e928257385 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.690866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9209ef-a2b2-4d25-b3bc-42e928257385 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.691017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9209ef-a2b2-4d25-b3bc-42e928257385 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.691253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9209ef-a2b2-4d25-b3bc-42e928257385 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.724884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1152/2294] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:21 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:21.731925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7e6cbc-3af3-44d4-9a68-d2cea5543220 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.768073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:21.776991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:21.784716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9209ef-a2b2-4d25-b3bc-42e928257385 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.786862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9209ef-a2b2-4d25-b3bc-42e928257385 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.799981 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1143/2295] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:21.820951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7e6cbc-3af3-44d4-9a68-d2cea5543220 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.827038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a997ed-2de0-4496-b8e7-ca2bb1566fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.827859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a997ed-2de0-4496-b8e7-ca2bb1566fb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.828231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a997ed-2de0-4496-b8e7-ca2bb1566fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.828574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a997ed-2de0-4496-b8e7-ca2bb1566fb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.829312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a997ed-2de0-4496-b8e7-ca2bb1566fb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.831464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a997ed-2de0-4496-b8e7-ca2bb1566fb2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:21.879041 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1153/2296] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:21 2022] POST /identity/v3/projects => generated 367 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:21.880196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9209ef-a2b2-4d25-b3bc-42e928257385 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:21.892695 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9209ef-a2b2-4d25-b3bc-42e928257385 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:21.962544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1ddfcb-c933-4205-8b57-7601bdecbfbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:21.963186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1ddfcb-c933-4205-8b57-7601bdecbfbc admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:21.963471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1ddfcb-c933-4205-8b57-7601bdecbfbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:21.963751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1ddfcb-c933-4205-8b57-7601bdecbfbc admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:21.964415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1ddfcb-c933-4205-8b57-7601bdecbfbc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:21.966381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1ddfcb-c933-4205-8b57-7601bdecbfbc admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:22.000264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1144/2297] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:21 2022] POST /identity/v3/users => generated 290 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:22.004974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a997ed-2de0-4496-b8e7-ca2bb1566fb2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:22.022303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a997ed-2de0-4496-b8e7-ca2bb1566fb2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:22.029781 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83010a02-8f78-41d7-9e76-25323ed1cff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:22.031082 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83010a02-8f78-41d7-9e76-25323ed1cff7 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:22.031579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83010a02-8f78-41d7-9e76-25323ed1cff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:22.032106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83010a02-8f78-41d7-9e76-25323ed1cff7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:22.033830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83010a02-8f78-41d7-9e76-25323ed1cff7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:22.036461 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83010a02-8f78-41d7-9e76-25323ed1cff7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:22.051272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1154/2298] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:22.057484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1ddfcb-c933-4205-8b57-7601bdecbfbc admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:22.076661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1ddfcb-c933-4205-8b57-7601bdecbfbc admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:22.086783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576f1d2b-3c45-4356-8664-d29e2b2f3d8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:22.087901 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f1d2b-3c45-4356-8664-d29e2b2f3d8c admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:22.088382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f1d2b-3c45-4356-8664-d29e2b2f3d8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:22.088849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f1d2b-3c45-4356-8664-d29e2b2f3d8c admin admin] PATH_INFO: `/v3/projects/2125b7f96c474cb19dba596e8c3028fc/users/969a11fc00874a0d9960b979fc06db1f/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:22.119369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576f1d2b-3c45-4356-8664-d29e2b2f3d8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2125b7f96c474cb19dba596e8c3028fc, user_id=969a11fc00874a0d9960b979fc06db1f, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:22.124728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576f1d2b-3c45-4356-8664-d29e2b2f3d8c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:22.153043 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1145/2299] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:22 2022] PUT /identity/v3/projects/2125b7f96c474cb19dba596e8c3028fc/users/969a11fc00874a0d9960b979fc06db1f/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:22.158555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83010a02-8f78-41d7-9e76-25323ed1cff7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:22.174227 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83010a02-8f78-41d7-9e76-25323ed1cff7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:22.302723 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac13e0b-8bc2-437e-af11-abb5968c9f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:22.303541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac13e0b-8bc2-437e-af11-abb5968c9f5f admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:22.303943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac13e0b-8bc2-437e-af11-abb5968c9f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:22.304339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac13e0b-8bc2-437e-af11-abb5968c9f5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:22.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac13e0b-8bc2-437e-af11-abb5968c9f5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:22.307522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac13e0b-8bc2-437e-af11-abb5968c9f5f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:22.324175 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1155/2300] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:22.329122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576f1d2b-3c45-4356-8664-d29e2b2f3d8c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:22.343965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576f1d2b-3c45-4356-8664-d29e2b2f3d8c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:22.351241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77164472-6a65-401f-b22b-865740c12bc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:22.352086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77164472-6a65-401f-b22b-865740c12bc5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:22.352445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77164472-6a65-401f-b22b-865740c12bc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:22.352788 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77164472-6a65-401f-b22b-865740c12bc5 admin admin] PATH_INFO: `/v3/projects/2125b7f96c474cb19dba596e8c3028fc/users/969a11fc00874a0d9960b979fc06db1f/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:22.355302 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77164472-6a65-401f-b22b-865740c12bc5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2125b7f96c474cb19dba596e8c3028fc, user_id=969a11fc00874a0d9960b979fc06db1f, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:22.358284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77164472-6a65-401f-b22b-865740c12bc5 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:22.372413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1146/2301] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:22 2022] PUT /identity/v3/projects/2125b7f96c474cb19dba596e8c3028fc/users/969a11fc00874a0d9960b979fc06db1f/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:22.383670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ea23c-adf7-475b-bbb8-45208a00c00c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:22.384005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ea23c-adf7-475b-bbb8-45208a00c00c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:22.384239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ea23c-adf7-475b-bbb8-45208a00c00c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:22.425593 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5a8ea23c-adf7-475b-bbb8-45208a00c00c None None] MFA Rules not processed for user `969a11fc00874a0d9960b979fc06db1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:22.427028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5a8ea23c-adf7-475b-bbb8-45208a00c00c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:22.509031 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1156/2302] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:22 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:22.522853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77164472-6a65-401f-b22b-865740c12bc5 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:22.602238 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77164472-6a65-401f-b22b-865740c12bc5 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:22.606674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99c4a55-c9b3-4bb4-89bc-30fd91e4937a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:22.607418 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99c4a55-c9b3-4bb4-89bc-30fd91e4937a service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:22.607832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99c4a55-c9b3-4bb4-89bc-30fd91e4937a service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:22.608201 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99c4a55-c9b3-4bb4-89bc-30fd91e4937a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:22.620005 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99c4a55-c9b3-4bb4-89bc-30fd91e4937a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:22.621609 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99c4a55-c9b3-4bb4-89bc-30fd91e4937a service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:22.633474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1147/2303] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:22 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:22.751644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8ea23c-adf7-475b-bbb8-45208a00c00c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:22.762690 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8ea23c-adf7-475b-bbb8-45208a00c00c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:22.766414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52dc63c-33e4-4cb0-8249-168fe3068698 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:22.767067 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dc63c-33e4-4cb0-8249-168fe3068698 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:22.767346 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dc63c-33e4-4cb0-8249-168fe3068698 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:22.768188 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dc63c-33e4-4cb0-8249-168fe3068698 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:22.796445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:22.848813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52dc63c-33e4-4cb0-8249-168fe3068698 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:22.850197 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52dc63c-33e4-4cb0-8249-168fe3068698 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:22.860571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1157/2304] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:22 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:24.614042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99c4a55-c9b3-4bb4-89bc-30fd91e4937a service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:24.693281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:24.696494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99c4a55-c9b3-4bb4-89bc-30fd91e4937a service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:24.701942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:24.702825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:24.703277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:24.703647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] PATH_INFO: `/v3/users/896c2d615013484da0a55a9dc8e4a65f` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:24.705917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] RBAC: Authorizing `identity:delete_user(user_id=896c2d615013484da0a55a9dc8e4a65f)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:24.708601 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:24.773516 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '896c2d615013484da0a55a9dc8e4a65f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:24.786428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '896c2d615013484da0a55a9dc8e4a65f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:24.787237 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '896c2d615013484da0a55a9dc8e4a65f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:24.787929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '896c2d615013484da0a55a9dc8e4a65f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:24.796617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '896c2d615013484da0a55a9dc8e4a65f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:24.819748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1148/2305] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:24 2022] DELETE /identity/v3/users/896c2d615013484da0a55a9dc8e4a65f => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:24.824844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792636b5-6673-4027-adf9-7fc29e84522a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:24.825716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792636b5-6673-4027-adf9-7fc29e84522a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:24.825716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792636b5-6673-4027-adf9-7fc29e84522a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:24.850729 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-792636b5-6673-4027-adf9-7fc29e84522a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:24.852216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-792636b5-6673-4027-adf9-7fc29e84522a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:24.939471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1158/2306] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:24.951463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:24.953054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:25.025133 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a633ab7-2270-47f0-8f8d-83da2abd254c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:25.029094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa99e8c-758a-4cf3-9400-7727c620aa75 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:25.029729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa99e8c-758a-4cf3-9400-7727c620aa75 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:25.030011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa99e8c-758a-4cf3-9400-7727c620aa75 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:25.030296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa99e8c-758a-4cf3-9400-7727c620aa75 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:25.041882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa99e8c-758a-4cf3-9400-7727c620aa75 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:25.043825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa99e8c-758a-4cf3-9400-7727c620aa75 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:25.057076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1149/2307] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:25.433878 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792636b5-6673-4027-adf9-7fc29e84522a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:25.436399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-792636b5-6673-4027-adf9-7fc29e84522a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:25.452859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792636b5-6673-4027-adf9-7fc29e84522a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:25.462662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:25.463907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:25.464522 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:25.465092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] PATH_INFO: `/v3/projects/240ea8be5fe94613b0d9ca70005d49df` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:25.467414 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] RBAC: Authorizing `identity:delete_project(project_id=240ea8be5fe94613b0d9ca70005d49df)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:25.471192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:25.530797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '240ea8be5fe94613b0d9ca70005d49df', 'request_id': 'req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:25.539061 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '240ea8be5fe94613b0d9ca70005d49df', 'request_id': 'req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:25.550870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '240ea8be5fe94613b0d9ca70005d49df', 'request_id': 'req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:25.565413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] The token cache is being invalidate because project 240ea8be5fe94613b0d9ca70005d49df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:25.565899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:25.566800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-2c2f5742-fff3-47d7-87b1-4073ca3a3dbd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:25.568423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1159/2308] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:25 2022] DELETE /identity/v3/projects/240ea8be5fe94613b0d9ca70005d49df => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:25.573718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16228129-bd26-4d27-b191-b7cf66562d57 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:25.574006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16228129-bd26-4d27-b191-b7cf66562d57 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:25.574281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16228129-bd26-4d27-b191-b7cf66562d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:25.598071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-16228129-bd26-4d27-b191-b7cf66562d57 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:25.599393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-16228129-bd26-4d27-b191-b7cf66562d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:25.668760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1150/2309] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:25.677566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec37981-0790-4677-9feb-b36309410a6f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:25.677855 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec37981-0790-4677-9feb-b36309410a6f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:25.678134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec37981-0790-4677-9feb-b36309410a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:25.701130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-fec37981-0790-4677-9feb-b36309410a6f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:25.702556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-fec37981-0790-4677-9feb-b36309410a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:25.715853 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1160/2310] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:25.721531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16228129-bd26-4d27-b191-b7cf66562d57 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:25.733873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16228129-bd26-4d27-b191-b7cf66562d57 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:25.740734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:25.741472 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:25.741944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:25.742407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:25.743342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:25.743705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:25.746186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:25.749108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:25.773016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1151/2311] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:48:25.777410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec37981-0790-4677-9feb-b36309410a6f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:25.789078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec37981-0790-4677-9feb-b36309410a6f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:25.793731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a64ea0-3333-4cd4-9f97-b7f011282024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:25.794362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a64ea0-3333-4cd4-9f97-b7f011282024 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:25.794679 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a64ea0-3333-4cd4-9f97-b7f011282024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:25.794953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a64ea0-3333-4cd4-9f97-b7f011282024 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:25.795612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a64ea0-3333-4cd4-9f97-b7f011282024 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:25.797567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a64ea0-3333-4cd4-9f97-b7f011282024 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:25.832806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1161/2312] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:25 2022] POST /identity/v3/projects => generated 369 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:25.839165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:25.854477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890695fa-8cc4-49cc-a8b0-97938b265cec admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:25.936397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b29c02-3459-4b59-8686-99946dd726a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:25.937259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b29c02-3459-4b59-8686-99946dd726a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:25.937711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b29c02-3459-4b59-8686-99946dd726a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:25.938174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b29c02-3459-4b59-8686-99946dd726a0 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:25.939323 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b29c02-3459-4b59-8686-99946dd726a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:25.943071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b29c02-3459-4b59-8686-99946dd726a0 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:25.981921 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1152/2313] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:25 2022] POST /identity/v3/users => generated 285 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:25.984641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a64ea0-3333-4cd4-9f97-b7f011282024 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:25.999784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a64ea0-3333-4cd4-9f97-b7f011282024 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:26.008109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f2c1d6-3f00-4a1c-8167-df9ebcdab0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:26.009124 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f2c1d6-3f00-4a1c-8167-df9ebcdab0a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:26.009503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f2c1d6-3f00-4a1c-8167-df9ebcdab0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:26.009987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f2c1d6-3f00-4a1c-8167-df9ebcdab0a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:26.010940 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f2c1d6-3f00-4a1c-8167-df9ebcdab0a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:26.013261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f2c1d6-3f00-4a1c-8167-df9ebcdab0a9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:26.029633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1162/2314] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:25 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:26.036766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b29c02-3459-4b59-8686-99946dd726a0 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:26.056235 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b29c02-3459-4b59-8686-99946dd726a0 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:26.065245 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10719678-26cf-41a6-b79a-79ae97b21e6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:26.066474 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10719678-26cf-41a6-b79a-79ae97b21e6d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:26.067097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10719678-26cf-41a6-b79a-79ae97b21e6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:26.067653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10719678-26cf-41a6-b79a-79ae97b21e6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:26.068952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10719678-26cf-41a6-b79a-79ae97b21e6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:26.072389 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10719678-26cf-41a6-b79a-79ae97b21e6d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:26.090876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1153/2315] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:26 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:26.097885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f2c1d6-3f00-4a1c-8167-df9ebcdab0a9 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:26.116273 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f2c1d6-3f00-4a1c-8167-df9ebcdab0a9 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:26.124926 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:26.126076 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:26.127458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:26.127979 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] PATH_INFO: `/v3/projects/81eba63c7ea8490d8a9a6600449267f4/users/2ad62e7a918e47848604ef80303d01cc/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:26.133625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:26.161640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81eba63c7ea8490d8a9a6600449267f4, user_id=2ad62e7a918e47848604ef80303d01cc, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:26.168583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:26.215431 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1163/2316] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:26 2022] PUT /identity/v3/projects/81eba63c7ea8490d8a9a6600449267f4/users/2ad62e7a918e47848604ef80303d01cc/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:26.233657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d383631-26fd-462a-924a-2e7ac49ec257 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:26.234036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d383631-26fd-462a-924a-2e7ac49ec257 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:26.234434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d383631-26fd-462a-924a-2e7ac49ec257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:26.294353 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-9d383631-26fd-462a-924a-2e7ac49ec257 None None] MFA Rules not processed for user `2ad62e7a918e47848604ef80303d01cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:26.295941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9d383631-26fd-462a-924a-2e7ac49ec257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:26.423230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1154/2317] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:26 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:26.439953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:26.442950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:26.575728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e48be63-3dac-45c2-ae04-f8c36523e9c1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:26.580604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c5e0ee-c34f-47fc-926d-035531db0994 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e3f6be584e34bfb9e430c7756d071d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:26.581314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c5e0ee-c34f-47fc-926d-035531db0994 service glance] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:26.581594 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c5e0ee-c34f-47fc-926d-035531db0994 service glance] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:26.581868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c5e0ee-c34f-47fc-926d-035531db0994 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:26.592718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c5e0ee-c34f-47fc-926d-035531db0994 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:26.594844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c5e0ee-c34f-47fc-926d-035531db0994 service glance] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:26.608373 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1164/2318] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:26 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:28.436783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d383631-26fd-462a-924a-2e7ac49ec257 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:28.438438 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-9d383631-26fd-462a-924a-2e7ac49ec257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:28.489942 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:28.557678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d383631-26fd-462a-924a-2e7ac49ec257 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:28.563297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:28.563957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:28.564766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:28.567985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] PATH_INFO: `/v3/users/f13e5850266448c095afbb3aa73b512b` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:28.569554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f13e5850266448c095afbb3aa73b512b)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:28.571729 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:28.648694 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f13e5850266448c095afbb3aa73b512b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:28.662073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f13e5850266448c095afbb3aa73b512b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:28.663386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f13e5850266448c095afbb3aa73b512b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:28.664338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f13e5850266448c095afbb3aa73b512b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:28.676717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f13e5850266448c095afbb3aa73b512b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:28.710001 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1155/2319] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:28 2022] DELETE /identity/v3/users/f13e5850266448c095afbb3aa73b512b => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:28.717558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a4393-6885-47d8-8eb8-baeadcd6a8e3 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:28.718139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a4393-6885-47d8-8eb8-baeadcd6a8e3 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:28.718643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a4393-6885-47d8-8eb8-baeadcd6a8e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:28.772497 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-194a4393-6885-47d8-8eb8-baeadcd6a8e3 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:28.774849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-194a4393-6885-47d8-8eb8-baeadcd6a8e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:28.871224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1165/2320] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:28.883646 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:28.885856 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:28.990093 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e418714-6bc2-4194-8baf-03339f107ca2 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:28.997095 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e41b48-cea3-4105-b5de-879d255c4bbe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:28.998230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e41b48-cea3-4105-b5de-879d255c4bbe service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:28.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e41b48-cea3-4105-b5de-879d255c4bbe service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:28.999224 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e41b48-cea3-4105-b5de-879d255c4bbe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:29.016654 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e41b48-cea3-4105-b5de-879d255c4bbe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:29.019914 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e41b48-cea3-4105-b5de-879d255c4bbe service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:29.038588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1156/2321] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:29.513958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194a4393-6885-47d8-8eb8-baeadcd6a8e3 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:29.516537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-194a4393-6885-47d8-8eb8-baeadcd6a8e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:29.545995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194a4393-6885-47d8-8eb8-baeadcd6a8e3 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:29.553360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:29.554448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:29.554971 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:29.555442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] PATH_INFO: `/v3/projects/e8a6e9bad74c4089b6c12b6e7d0cd1a6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:29.557995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8a6e9bad74c4089b6c12b6e7d0cd1a6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:29.561888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:29.629777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:29.637826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8a6e9bad74c4089b6c12b6e7d0cd1a6', 'request_id': 'req-a461c9c0-35fa-454d-9973-20df529252f0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.647508 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8a6e9bad74c4089b6c12b6e7d0cd1a6', 'request_id': 'req-a461c9c0-35fa-454d-9973-20df529252f0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.660326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8a6e9bad74c4089b6c12b6e7d0cd1a6', 'request_id': 'req-a461c9c0-35fa-454d-9973-20df529252f0'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] The token cache is being invalidate because project e8a6e9bad74c4089b6c12b6e7d0cd1a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:29.675364 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.676108 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.677818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1166/2322] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:29 2022] DELETE /identity/v3/projects/e8a6e9bad74c4089b6c12b6e7d0cd1a6 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:29.684062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e41b48-cea3-4105-b5de-879d255c4bbe service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:29.701734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e41b48-cea3-4105-b5de-879d255c4bbe service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:29.748090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:29.788036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:29.788649 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:29.788973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:29.789290 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] PATH_INFO: `/v3/users/969a11fc00874a0d9960b979fc06db1f` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:29.790559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] RBAC: Authorizing `identity:delete_user(user_id=969a11fc00874a0d9960b979fc06db1f)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:29.792646 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:29.861220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '969a11fc00874a0d9960b979fc06db1f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.871641 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '969a11fc00874a0d9960b979fc06db1f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.872497 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '969a11fc00874a0d9960b979fc06db1f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.873247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '969a11fc00874a0d9960b979fc06db1f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.884243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-f82e727f-34d8-426d-897f-deb0f7d55600 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '969a11fc00874a0d9960b979fc06db1f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:29.906507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1157/2323] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:29 2022] DELETE /identity/v3/users/969a11fc00874a0d9960b979fc06db1f => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:30.298324 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:30.301443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:30.326975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a461c9c0-35fa-454d-9973-20df529252f0 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:30.445443 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:30.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:30.446387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:30.446706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] PATH_INFO: `/v3/projects/2125b7f96c474cb19dba596e8c3028fc` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:30.448180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2125b7f96c474cb19dba596e8c3028fc)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:30.450150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:30.566065 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2125b7f96c474cb19dba596e8c3028fc', 'request_id': 'req-d58004a0-b742-4b69-9c24-f90f26f0b9e7'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:30.576152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2125b7f96c474cb19dba596e8c3028fc', 'request_id': 'req-d58004a0-b742-4b69-9c24-f90f26f0b9e7'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:30.586849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2125b7f96c474cb19dba596e8c3028fc', 'request_id': 'req-d58004a0-b742-4b69-9c24-f90f26f0b9e7'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:30.602684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] The token cache is being invalidate because project 2125b7f96c474cb19dba596e8c3028fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:30.602974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:30.603694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-d58004a0-b742-4b69-9c24-f90f26f0b9e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:30.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1167/2324] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:30 2022] DELETE /identity/v3/projects/2125b7f96c474cb19dba596e8c3028fc => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:30.611707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c4491-2971-46be-b9bc-ff6bdf7c6407 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:30.612657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c4491-2971-46be-b9bc-ff6bdf7c6407 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:30.612657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c4491-2971-46be-b9bc-ff6bdf7c6407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:30.678058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a99c4491-2971-46be-b9bc-ff6bdf7c6407 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:30.679369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a99c4491-2971-46be-b9bc-ff6bdf7c6407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:30.774317 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1158/2325] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:30.785212 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed66c037-06f9-45b1-bdb6-88198481eeb1 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:30.785498 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed66c037-06f9-45b1-bdb6-88198481eeb1 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:30.785773 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed66c037-06f9-45b1-bdb6-88198481eeb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:30.809419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ed66c037-06f9-45b1-bdb6-88198481eeb1 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:30.810765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ed66c037-06f9-45b1-bdb6-88198481eeb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:30.820458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1168/2326] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:30.825273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c4491-2971-46be-b9bc-ff6bdf7c6407 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:30.836997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c4491-2971-46be-b9bc-ff6bdf7c6407 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:30.842446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:30.843129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:30.843410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:30.843686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:30.844624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:30.844935 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:30.847826 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:30.862793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1159/2327] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:48:30.867180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed66c037-06f9-45b1-bdb6-88198481eeb1 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:30.878760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed66c037-06f9-45b1-bdb6-88198481eeb1 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:30.883968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb72cb-376a-41a4-b62a-c7deac4ec373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:30.884625 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb72cb-376a-41a4-b62a-c7deac4ec373 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:30.884904 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb72cb-376a-41a4-b62a-c7deac4ec373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:30.885173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb72cb-376a-41a4-b62a-c7deac4ec373 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:30.885837 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cb72cb-376a-41a4-b62a-c7deac4ec373 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:30.887840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cb72cb-376a-41a4-b62a-c7deac4ec373 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:30.921974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1169/2328] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:30 2022] POST /identity/v3/projects => generated 379 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:30.928651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:30.947835 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eede5e4-0c82-4499-b67d-a85488fddf70 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:30.967495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:31.052543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f603ed-fbd1-4394-a429-abc3b15d34ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.053605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f603ed-fbd1-4394-a429-abc3b15d34ae admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.054084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f603ed-fbd1-4394-a429-abc3b15d34ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.054580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f603ed-fbd1-4394-a429-abc3b15d34ae admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.055776 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f603ed-fbd1-4394-a429-abc3b15d34ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.059447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f603ed-fbd1-4394-a429-abc3b15d34ae admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.092732 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1160/2329] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:30 2022] POST /identity/v3/users => generated 290 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:31.098963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb72cb-376a-41a4-b62a-c7deac4ec373 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.111675 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb72cb-376a-41a4-b62a-c7deac4ec373 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.117987 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b45291d-fa81-44e1-9eca-0465df3a58d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.118665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b45291d-fa81-44e1-9eca-0465df3a58d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.118945 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b45291d-fa81-44e1-9eca-0465df3a58d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.119223 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b45291d-fa81-44e1-9eca-0465df3a58d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.119886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b45291d-fa81-44e1-9eca-0465df3a58d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.121554 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b45291d-fa81-44e1-9eca-0465df3a58d9 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.134677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1170/2330] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:31.139187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f603ed-fbd1-4394-a429-abc3b15d34ae admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.153152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f603ed-fbd1-4394-a429-abc3b15d34ae admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.158448 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ab681d-8b69-4391-846c-b316d85696ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.159121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab681d-8b69-4391-846c-b316d85696ee admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.159342 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab681d-8b69-4391-846c-b316d85696ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.159651 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab681d-8b69-4391-846c-b316d85696ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.160356 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ab681d-8b69-4391-846c-b316d85696ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.162085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ab681d-8b69-4391-846c-b316d85696ee admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.176847 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1161/2331] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:31.181976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b45291d-fa81-44e1-9eca-0465df3a58d9 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.200876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b45291d-fa81-44e1-9eca-0465df3a58d9 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.207997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8554df-6c4f-4628-91db-8a3f9a5981d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.208704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8554df-6c4f-4628-91db-8a3f9a5981d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.208985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8554df-6c4f-4628-91db-8a3f9a5981d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.209261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8554df-6c4f-4628-91db-8a3f9a5981d5 admin admin] PATH_INFO: `/v3/projects/1300da1f10cc4f6e87670c5263bdc1dc/users/42f7d77dca3840b7ada9be0cb0d2162f/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.229921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8554df-6c4f-4628-91db-8a3f9a5981d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1300da1f10cc4f6e87670c5263bdc1dc, user_id=42f7d77dca3840b7ada9be0cb0d2162f, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.233732 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8554df-6c4f-4628-91db-8a3f9a5981d5 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.247347 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:31.255620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1171/2332] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:31 2022] PUT /identity/v3/projects/1300da1f10cc4f6e87670c5263bdc1dc/users/42f7d77dca3840b7ada9be0cb0d2162f/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:31.268060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b194997-c01e-4f0b-80b4-11073c12f7db None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.268396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b194997-c01e-4f0b-80b4-11073c12f7db None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.268625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b194997-c01e-4f0b-80b4-11073c12f7db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.304363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2b194997-c01e-4f0b-80b4-11073c12f7db None None] MFA Rules not processed for user `42f7d77dca3840b7ada9be0cb0d2162f`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:31.305705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2b194997-c01e-4f0b-80b4-11073c12f7db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:31.406525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1162/2333] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:31.412850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8554df-6c4f-4628-91db-8a3f9a5981d5 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.500730 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8554df-6c4f-4628-91db-8a3f9a5981d5 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.506980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8325823-0e5a-4f31-aa9d-a1e0e000ee4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.507610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8325823-0e5a-4f31-aa9d-a1e0e000ee4f admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.507883 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8325823-0e5a-4f31-aa9d-a1e0e000ee4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.508155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8325823-0e5a-4f31-aa9d-a1e0e000ee4f admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.508796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8325823-0e5a-4f31-aa9d-a1e0e000ee4f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.510748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8325823-0e5a-4f31-aa9d-a1e0e000ee4f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.547574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1172/2334] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:31 2022] POST /identity/v3/projects => generated 379 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:31.556341 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b194997-c01e-4f0b-80b4-11073c12f7db None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.572058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b194997-c01e-4f0b-80b4-11073c12f7db None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.662251 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af0dd44-0abe-4ac0-866b-ffc34813e788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.662876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0dd44-0abe-4ac0-866b-ffc34813e788 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.663222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0dd44-0abe-4ac0-866b-ffc34813e788 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.663559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0dd44-0abe-4ac0-866b-ffc34813e788 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.664277 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af0dd44-0abe-4ac0-866b-ffc34813e788 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.666355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af0dd44-0abe-4ac0-866b-ffc34813e788 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.700284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1163/2335] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:31 2022] POST /identity/v3/users => generated 296 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:31.706255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8325823-0e5a-4f31-aa9d-a1e0e000ee4f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.723117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8325823-0e5a-4f31-aa9d-a1e0e000ee4f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.729036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fdc207-cf2a-4b37-9024-f17455b30ee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.730474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fdc207-cf2a-4b37-9024-f17455b30ee5 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.730474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fdc207-cf2a-4b37-9024-f17455b30ee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.730474 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fdc207-cf2a-4b37-9024-f17455b30ee5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.731432 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fdc207-cf2a-4b37-9024-f17455b30ee5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.733213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fdc207-cf2a-4b37-9024-f17455b30ee5 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.754502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1173/2336] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:31.761019 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af0dd44-0abe-4ac0-866b-ffc34813e788 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.776860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af0dd44-0abe-4ac0-866b-ffc34813e788 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.782432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904076ce-09aa-4c0b-8379-db5d0add5f63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.783289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904076ce-09aa-4c0b-8379-db5d0add5f63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.783625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904076ce-09aa-4c0b-8379-db5d0add5f63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.783938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904076ce-09aa-4c0b-8379-db5d0add5f63 admin admin] PATH_INFO: `/v3/projects/7a51f934cdc84297bf17e1e71182e4f3/users/a1da4f2dc7504292856b1020db6ad1fc/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.808800 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:31.818143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904076ce-09aa-4c0b-8379-db5d0add5f63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a51f934cdc84297bf17e1e71182e4f3, user_id=a1da4f2dc7504292856b1020db6ad1fc, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.824468 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904076ce-09aa-4c0b-8379-db5d0add5f63 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.851668 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1164/2337] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:31 2022] PUT /identity/v3/projects/7a51f934cdc84297bf17e1e71182e4f3/users/a1da4f2dc7504292856b1020db6ad1fc/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:31.857665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fdc207-cf2a-4b37-9024-f17455b30ee5 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.872094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fdc207-cf2a-4b37-9024-f17455b30ee5 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.949372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad11127-72ca-46f2-b355-711930d1b225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.950056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad11127-72ca-46f2-b355-711930d1b225 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.950376 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad11127-72ca-46f2-b355-711930d1b225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.950739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad11127-72ca-46f2-b355-711930d1b225 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.951436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad11127-72ca-46f2-b355-711930d1b225 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.953286 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad11127-72ca-46f2-b355-711930d1b225 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:31.966610 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1174/2338] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:31.973107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904076ce-09aa-4c0b-8379-db5d0add5f63 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:31.984176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904076ce-09aa-4c0b-8379-db5d0add5f63 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:31.989571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:31.990275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:31.990584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:31.990858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] PATH_INFO: `/v3/projects/7a51f934cdc84297bf17e1e71182e4f3/users/a1da4f2dc7504292856b1020db6ad1fc/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:31.992852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a51f934cdc84297bf17e1e71182e4f3, user_id=a1da4f2dc7504292856b1020db6ad1fc, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:31.995836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:32.010363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1165/2339] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:31 2022] PUT /identity/v3/projects/7a51f934cdc84297bf17e1e71182e4f3/users/a1da4f2dc7504292856b1020db6ad1fc/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:32.022357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3e39ee-5bbf-4705-98ed-96778258ab3d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:32.022736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3e39ee-5bbf-4705-98ed-96778258ab3d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:32.023136 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3e39ee-5bbf-4705-98ed-96778258ab3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:32.060899 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-3f3e39ee-5bbf-4705-98ed-96778258ab3d None None] MFA Rules not processed for user `a1da4f2dc7504292856b1020db6ad1fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:32.062473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3f3e39ee-5bbf-4705-98ed-96778258ab3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:32.143685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:32.156565 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1175/2340] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:32 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:32.175029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:32.177132 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:32.263913 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63316e40-5a9b-4074-a109-ce848e62be04 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:32.271442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff86d77-b15f-4473-96ac-6369d1d1448a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:32.272478 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff86d77-b15f-4473-96ac-6369d1d1448a service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:32.272920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff86d77-b15f-4473-96ac-6369d1d1448a service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:32.273412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff86d77-b15f-4473-96ac-6369d1d1448a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:32.287866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff86d77-b15f-4473-96ac-6369d1d1448a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:32.289528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff86d77-b15f-4473-96ac-6369d1d1448a service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:32.303406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1166/2341] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:32 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:32.819157 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3e39ee-5bbf-4705-98ed-96778258ab3d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:32.916232 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3e39ee-5bbf-4705-98ed-96778258ab3d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:32.922275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:32.923130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:32.923569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:32.924555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] PATH_INFO: `/v3/users/42f7d77dca3840b7ada9be0cb0d2162f` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:32.925716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] RBAC: Authorizing `identity:delete_user(user_id=42f7d77dca3840b7ada9be0cb0d2162f)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:32.928493 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:33.010523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42f7d77dca3840b7ada9be0cb0d2162f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.026013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42f7d77dca3840b7ada9be0cb0d2162f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.029752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42f7d77dca3840b7ada9be0cb0d2162f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.030983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42f7d77dca3840b7ada9be0cb0d2162f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.042477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42f7d77dca3840b7ada9be0cb0d2162f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.075255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1176/2342] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:32 2022] DELETE /identity/v3/users/42f7d77dca3840b7ada9be0cb0d2162f => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:33.081806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528c0417-f418-4793-b718-ee226d1506dc None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:33.082882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528c0417-f418-4793-b718-ee226d1506dc None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:33.082882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528c0417-f418-4793-b718-ee226d1506dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:33.110792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-528c0417-f418-4793-b718-ee226d1506dc None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:33.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-528c0417-f418-4793-b718-ee226d1506dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:33.204023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1167/2343] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:33.218833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:33.221557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:33.316033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a49748-469b-47b3-a3fb-ebf4cba6b042 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:33.320518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dbe49b-9ede-4eaa-b97d-b7738c8d4473 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:33.321282 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dbe49b-9ede-4eaa-b97d-b7738c8d4473 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:33.321737 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dbe49b-9ede-4eaa-b97d-b7738c8d4473 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:33.322220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dbe49b-9ede-4eaa-b97d-b7738c8d4473 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:33.335544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3dbe49b-9ede-4eaa-b97d-b7738c8d4473 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:33.336907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3dbe49b-9ede-4eaa-b97d-b7738c8d4473 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:33.348463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1177/2344] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:33.772974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528c0417-f418-4793-b718-ee226d1506dc None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:33.774232 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-528c0417-f418-4793-b718-ee226d1506dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:33.793364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528c0417-f418-4793-b718-ee226d1506dc None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:33.801218 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:33.802051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:33.802459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:33.802868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] PATH_INFO: `/v3/projects/1300da1f10cc4f6e87670c5263bdc1dc` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:33.805316 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1300da1f10cc4f6e87670c5263bdc1dc)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:33.815740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:33.906523 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1300da1f10cc4f6e87670c5263bdc1dc', 'request_id': 'req-14f04371-833a-45e8-b7f9-c870ea6b78b7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.915442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1300da1f10cc4f6e87670c5263bdc1dc', 'request_id': 'req-14f04371-833a-45e8-b7f9-c870ea6b78b7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.924656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1300da1f10cc4f6e87670c5263bdc1dc', 'request_id': 'req-14f04371-833a-45e8-b7f9-c870ea6b78b7'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.944672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] The token cache is being invalidate because project 1300da1f10cc4f6e87670c5263bdc1dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:33.945181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.951041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:33.951041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1168/2345] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:33 2022] DELETE /identity/v3/projects/1300da1f10cc4f6e87670c5263bdc1dc => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:33.953938 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dbe49b-9ede-4eaa-b97d-b7738c8d4473 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:33.972909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dbe49b-9ede-4eaa-b97d-b7738c8d4473 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:33.988396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:34.065784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:34.066441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:34.066756 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:34.067270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] PATH_INFO: `/v3/users/a1da4f2dc7504292856b1020db6ad1fc` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:34.068525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1da4f2dc7504292856b1020db6ad1fc)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:34.070596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:34.155786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1da4f2dc7504292856b1020db6ad1fc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:34.165504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1da4f2dc7504292856b1020db6ad1fc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:34.166179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1da4f2dc7504292856b1020db6ad1fc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:34.166848 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1da4f2dc7504292856b1020db6ad1fc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:34.180996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-78037fb0-26da-4b0e-ae23-6c3a5110b460 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1da4f2dc7504292856b1020db6ad1fc'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:34.209164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1178/2346] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:33 2022] DELETE /identity/v3/users/a1da4f2dc7504292856b1020db6ad1fc => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:34.727118 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:34.729723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:34.751432 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:34.761812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f04371-833a-45e8-b7f9-c870ea6b78b7 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:34.889994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:34.891940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:34.891940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:34.893073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] PATH_INFO: `/v3/projects/7a51f934cdc84297bf17e1e71182e4f3` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:34.896188 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a51f934cdc84297bf17e1e71182e4f3)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:34.900615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:35.001152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a51f934cdc84297bf17e1e71182e4f3', 'request_id': 'req-665820ca-12d2-4402-9c17-e9f391c23079'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:35.009528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a51f934cdc84297bf17e1e71182e4f3', 'request_id': 'req-665820ca-12d2-4402-9c17-e9f391c23079'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:35.018123 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a51f934cdc84297bf17e1e71182e4f3', 'request_id': 'req-665820ca-12d2-4402-9c17-e9f391c23079'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:35.040242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] The token cache is being invalidate because project 7a51f934cdc84297bf17e1e71182e4f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:35.040706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:35.042079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-665820ca-12d2-4402-9c17-e9f391c23079 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:35.044040 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1169/2347] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:34 2022] DELETE /identity/v3/projects/7a51f934cdc84297bf17e1e71182e4f3 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:35.052219 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc6bd71-ac81-4e99-bf43-10dc80816e83 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.054050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc6bd71-ac81-4e99-bf43-10dc80816e83 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.054731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc6bd71-ac81-4e99-bf43-10dc80816e83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.084458 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-bcc6bd71-ac81-4e99-bf43-10dc80816e83 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:35.086137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-bcc6bd71-ac81-4e99-bf43-10dc80816e83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:35.182584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1179/2348] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:35.192220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73589bf-1168-4c2a-baf5-4c024b981899 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.192560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73589bf-1168-4c2a-baf5-4c024b981899 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.192780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73589bf-1168-4c2a-baf5-4c024b981899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.217154 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e73589bf-1168-4c2a-baf5-4c024b981899 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:35.218454 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e73589bf-1168-4c2a-baf5-4c024b981899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:35.228217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1170/2349] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:35.235961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc6bd71-ac81-4e99-bf43-10dc80816e83 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:35.250323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc6bd71-ac81-4e99-bf43-10dc80816e83 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:35.258639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:35.259255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.259603 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.260133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.260914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:35.261320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:35.263189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:35.279641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1180/2350] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:48:35.288345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73589bf-1168-4c2a-baf5-4c024b981899 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:35.301825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73589bf-1168-4c2a-baf5-4c024b981899 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:35.307229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7634c6e-9442-4420-9f2f-0397a0e878c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:35.307909 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7634c6e-9442-4420-9f2f-0397a0e878c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.308343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7634c6e-9442-4420-9f2f-0397a0e878c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.308759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7634c6e-9442-4420-9f2f-0397a0e878c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.309558 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7634c6e-9442-4420-9f2f-0397a0e878c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:35.312462 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7634c6e-9442-4420-9f2f-0397a0e878c5 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:35.350250 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1171/2351] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:35 2022] POST /identity/v3/projects => generated 383 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:35.356836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:35.375502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d50398-1601-44ff-bb14-6c32c09394bf admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:35.409102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:35.469502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1318b0da-d4b1-48ba-bb12-3480857f2742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:35.471477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1318b0da-d4b1-48ba-bb12-3480857f2742 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.471477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1318b0da-d4b1-48ba-bb12-3480857f2742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.471477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1318b0da-d4b1-48ba-bb12-3480857f2742 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.472030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1318b0da-d4b1-48ba-bb12-3480857f2742 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:35.474154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1318b0da-d4b1-48ba-bb12-3480857f2742 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:35.501512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1181/2352] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:35 2022] POST /identity/v3/users => generated 292 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:35.506983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7634c6e-9442-4420-9f2f-0397a0e878c5 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:35.526920 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7634c6e-9442-4420-9f2f-0397a0e878c5 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:35.535405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076eec16-9851-4e24-b885-940da74bb5dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:35.537072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076eec16-9851-4e24-b885-940da74bb5dc admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.537306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076eec16-9851-4e24-b885-940da74bb5dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.537947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076eec16-9851-4e24-b885-940da74bb5dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.539489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076eec16-9851-4e24-b885-940da74bb5dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:35.550049 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076eec16-9851-4e24-b885-940da74bb5dc admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:35.566625 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1172/2353] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:35 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:35.571593 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1318b0da-d4b1-48ba-bb12-3480857f2742 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:35.588506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1318b0da-d4b1-48ba-bb12-3480857f2742 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:35.597069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2db9b2-5457-4f2c-a14d-f839eefb167b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:35.598111 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2db9b2-5457-4f2c-a14d-f839eefb167b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.598536 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2db9b2-5457-4f2c-a14d-f839eefb167b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.598914 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2db9b2-5457-4f2c-a14d-f839eefb167b admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.599731 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2db9b2-5457-4f2c-a14d-f839eefb167b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:35.602063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2db9b2-5457-4f2c-a14d-f839eefb167b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:35.619802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1182/2354] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:35 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:35.625597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076eec16-9851-4e24-b885-940da74bb5dc admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:35.640572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076eec16-9851-4e24-b885-940da74bb5dc admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:35.648393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4955499b-31a1-4a6b-8a20-028c1449bd1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:35.649091 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4955499b-31a1-4a6b-8a20-028c1449bd1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.649384 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4955499b-31a1-4a6b-8a20-028c1449bd1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.649638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4955499b-31a1-4a6b-8a20-028c1449bd1f admin admin] PATH_INFO: `/v3/projects/d661befea1d84f4ea42b6f37c54bed50/users/d5d00369644e495f81a78abbd213ac74/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.680167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4955499b-31a1-4a6b-8a20-028c1449bd1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=d661befea1d84f4ea42b6f37c54bed50, user_id=d5d00369644e495f81a78abbd213ac74, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:35.685830 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4955499b-31a1-4a6b-8a20-028c1449bd1f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:35.710082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1173/2355] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:35 2022] PUT /identity/v3/projects/d661befea1d84f4ea42b6f37c54bed50/users/d5d00369644e495f81a78abbd213ac74/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:35.729879 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5989818f-a874-4756-b7f3-9a86d3a5cdb2 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.730163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5989818f-a874-4756-b7f3-9a86d3a5cdb2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.730424 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5989818f-a874-4756-b7f3-9a86d3a5cdb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.765980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5989818f-a874-4756-b7f3-9a86d3a5cdb2 None None] MFA Rules not processed for user `d5d00369644e495f81a78abbd213ac74`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:35.768181 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5989818f-a874-4756-b7f3-9a86d3a5cdb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:35.850697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1183/2356] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:35.855528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4955499b-31a1-4a6b-8a20-028c1449bd1f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:35.908752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:35.954155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4955499b-31a1-4a6b-8a20-028c1449bd1f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:35.959273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e7220f-e438-4816-b237-5efaec1dce6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:35.959906 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7220f-e438-4816-b237-5efaec1dce6d admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:35.960219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7220f-e438-4816-b237-5efaec1dce6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:35.960492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7220f-e438-4816-b237-5efaec1dce6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:35.961143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e7220f-e438-4816-b237-5efaec1dce6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:35.963798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e7220f-e438-4816-b237-5efaec1dce6d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:36.006517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1174/2357] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:35 2022] POST /identity/v3/projects => generated 383 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:36.012952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5989818f-a874-4756-b7f3-9a86d3a5cdb2 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:36.032641 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5989818f-a874-4756-b7f3-9a86d3a5cdb2 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:36.126319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab57f01-8225-426f-b680-15a577c7b506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:36.127677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab57f01-8225-426f-b680-15a577c7b506 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:36.128045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab57f01-8225-426f-b680-15a577c7b506 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:36.128538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab57f01-8225-426f-b680-15a577c7b506 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:36.129640 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab57f01-8225-426f-b680-15a577c7b506 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:36.132261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab57f01-8225-426f-b680-15a577c7b506 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:36.162134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1184/2358] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:36 2022] POST /identity/v3/users => generated 298 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:36.166908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e7220f-e438-4816-b237-5efaec1dce6d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:36.180141 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e7220f-e438-4816-b237-5efaec1dce6d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:36.186127 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b20e993-9ee4-4417-a55d-efd54525b3b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:36.186833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b20e993-9ee4-4417-a55d-efd54525b3b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:36.187109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b20e993-9ee4-4417-a55d-efd54525b3b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:36.187381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b20e993-9ee4-4417-a55d-efd54525b3b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:36.188053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b20e993-9ee4-4417-a55d-efd54525b3b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:36.190427 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b20e993-9ee4-4417-a55d-efd54525b3b8 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:36.203253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1175/2359] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:36 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:48:36.208526 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab57f01-8225-426f-b680-15a577c7b506 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:36.222384 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab57f01-8225-426f-b680-15a577c7b506 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:36.228552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0c2dd8-a735-472c-a64e-7244fd1e8654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:36.229353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c2dd8-a735-472c-a64e-7244fd1e8654 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:36.229669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c2dd8-a735-472c-a64e-7244fd1e8654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:36.229950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c2dd8-a735-472c-a64e-7244fd1e8654 admin admin] PATH_INFO: `/v3/projects/7099aeed78114a77b08c019d4bb01a78/users/2cf142b1ede649e1bfdf05a6b3196e0a/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:36.249758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0c2dd8-a735-472c-a64e-7244fd1e8654 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7099aeed78114a77b08c019d4bb01a78, user_id=2cf142b1ede649e1bfdf05a6b3196e0a, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:36.253033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0c2dd8-a735-472c-a64e-7244fd1e8654 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:36.256798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:36.270980 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1185/2360] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:36 2022] PUT /identity/v3/projects/7099aeed78114a77b08c019d4bb01a78/users/2cf142b1ede649e1bfdf05a6b3196e0a/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:36.275272 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b20e993-9ee4-4417-a55d-efd54525b3b8 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:36.288186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b20e993-9ee4-4417-a55d-efd54525b3b8 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:36.362719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e8c60c-75fb-4fd2-baa8-3c3d8d93bda8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:36.363380 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e8c60c-75fb-4fd2-baa8-3c3d8d93bda8 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:36.363654 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e8c60c-75fb-4fd2-baa8-3c3d8d93bda8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:36.363925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e8c60c-75fb-4fd2-baa8-3c3d8d93bda8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:36.364586 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e8c60c-75fb-4fd2-baa8-3c3d8d93bda8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:36.366425 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e8c60c-75fb-4fd2-baa8-3c3d8d93bda8 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:36.379021 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1176/2361] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:36 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:36.384544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0c2dd8-a735-472c-a64e-7244fd1e8654 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:36.403786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0c2dd8-a735-472c-a64e-7244fd1e8654 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:36.412476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:36.413604 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:36.414084 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:36.414614 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] PATH_INFO: `/v3/projects/7099aeed78114a77b08c019d4bb01a78/users/2cf142b1ede649e1bfdf05a6b3196e0a/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:36.418186 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7099aeed78114a77b08c019d4bb01a78, user_id=2cf142b1ede649e1bfdf05a6b3196e0a, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:36.423744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:36.441804 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1186/2362] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:36 2022] PUT /identity/v3/projects/7099aeed78114a77b08c019d4bb01a78/users/2cf142b1ede649e1bfdf05a6b3196e0a/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:36.517964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8383d36-31f9-444a-a4e4-e87b2d79da07 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:36.518268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8383d36-31f9-444a-a4e4-e87b2d79da07 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:36.518540 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8383d36-31f9-444a-a4e4-e87b2d79da07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:36.552769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c8383d36-31f9-444a-a4e4-e87b2d79da07 None None] MFA Rules not processed for user `2cf142b1ede649e1bfdf05a6b3196e0a`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:36.554081 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c8383d36-31f9-444a-a4e4-e87b2d79da07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:36.669080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1177/2363] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:36 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:36.682333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:36.684369 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:36.793647 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864f6bbe-5c1d-4dc7-a327-425391edf138 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:36.798052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:36.798967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:36.799387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:36.800225 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:36.881903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:36.884985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:36.905735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1187/2364] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:41.639343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8383d36-31f9-444a-a4e4-e87b2d79da07 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:41.650619 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8383d36-31f9-444a-a4e4-e87b2d79da07 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:41.654343 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bab579-a3e4-49a2-8912-965eee7563b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:41.655026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bab579-a3e4-49a2-8912-965eee7563b1 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:41.655306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bab579-a3e4-49a2-8912-965eee7563b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:41.655575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bab579-a3e4-49a2-8912-965eee7563b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:41.666604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bab579-a3e4-49a2-8912-965eee7563b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:41.668079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bab579-a3e4-49a2-8912-965eee7563b1 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:41.680171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1178/2365] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:41 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:41.842077 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:41.844297 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:42.006358 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c39d396-9df0-4eb0-8074-bddf0f6f9f95 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:42.014007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd85bceb-3f39-446f-b5ee-321e26f8195d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:42.014967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85bceb-3f39-446f-b5ee-321e26f8195d service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:42.015305 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85bceb-3f39-446f-b5ee-321e26f8195d service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:42.015575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85bceb-3f39-446f-b5ee-321e26f8195d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:42.020357 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-bd85bceb-3f39-446f-b5ee-321e26f8195d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:42.046208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd85bceb-3f39-446f-b5ee-321e26f8195d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:42.049930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd85bceb-3f39-446f-b5ee-321e26f8195d service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:42.072748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1188/2366] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:49.062707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bab579-a3e4-49a2-8912-965eee7563b1 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:49.087026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:49.167450 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bab579-a3e4-49a2-8912-965eee7563b1 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:49.174121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:49.175055 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:49.175461 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:49.175836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] PATH_INFO: `/v3/users/d5d00369644e495f81a78abbd213ac74` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:49.177149 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d5d00369644e495f81a78abbd213ac74)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:49.179268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:49.238254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd5d00369644e495f81a78abbd213ac74'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:49.247709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd5d00369644e495f81a78abbd213ac74'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:49.248617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd5d00369644e495f81a78abbd213ac74'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:49.249331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd5d00369644e495f81a78abbd213ac74'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:49.259699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd5d00369644e495f81a78abbd213ac74'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:49.281045 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1179/2367] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:49 2022] DELETE /identity/v3/users/d5d00369644e495f81a78abbd213ac74 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:49.287250 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f0f489-2bc0-427b-9651-26264010c6e6 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:49.287872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f0f489-2bc0-427b-9651-26264010c6e6 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:49.287872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f0f489-2bc0-427b-9651-26264010c6e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:49.304502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:49.311415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e3f0f489-2bc0-427b-9651-26264010c6e6 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:49.313335 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e3f0f489-2bc0-427b-9651-26264010c6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:49.389733 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1189/2368] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:49.404687 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:49.406644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:49.527476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a5181-6242-44e3-8755-6c4c60e503b2 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:49.533739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cddb3-c11a-4c5c-b7b8-a409c3a3880f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:49.534740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cddb3-c11a-4c5c-b7b8-a409c3a3880f service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:49.535217 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cddb3-c11a-4c5c-b7b8-a409c3a3880f service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:49.535698 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cddb3-c11a-4c5c-b7b8-a409c3a3880f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:49.553181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7cddb3-c11a-4c5c-b7b8-a409c3a3880f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:49.556035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7cddb3-c11a-4c5c-b7b8-a409c3a3880f service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:49.575479 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1180/2369] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:48:50.206984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f0f489-2bc0-427b-9651-26264010c6e6 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:50.209689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e3f0f489-2bc0-427b-9651-26264010c6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:50.249313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f0f489-2bc0-427b-9651-26264010c6e6 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:50.256859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:50.257950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:50.258167 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:50.258951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] PATH_INFO: `/v3/projects/d661befea1d84f4ea42b6f37c54bed50` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:50.261037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d661befea1d84f4ea42b6f37c54bed50)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:50.263507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:50.323319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd661befea1d84f4ea42b6f37c54bed50', 'request_id': 'req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.330661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd661befea1d84f4ea42b6f37c54bed50', 'request_id': 'req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.339919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd661befea1d84f4ea42b6f37c54bed50', 'request_id': 'req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.354203 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] The token cache is being invalidate because project d661befea1d84f4ea42b6f37c54bed50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:50.354585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.355295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1190/2370] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:50 2022] DELETE /identity/v3/projects/d661befea1d84f4ea42b6f37c54bed50 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:50.363529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cddb3-c11a-4c5c-b7b8-a409c3a3880f service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:50.375053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cddb3-c11a-4c5c-b7b8-a409c3a3880f service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:50.389256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:50.463989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:50.464674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:50.464991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:50.465295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] PATH_INFO: `/v3/users/2cf142b1ede649e1bfdf05a6b3196e0a` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:50.466610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2cf142b1ede649e1bfdf05a6b3196e0a)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:50.468670 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:50.527061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2cf142b1ede649e1bfdf05a6b3196e0a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.535778 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2cf142b1ede649e1bfdf05a6b3196e0a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.536858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2cf142b1ede649e1bfdf05a6b3196e0a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.537725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2cf142b1ede649e1bfdf05a6b3196e0a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.548707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d24f13bb-4a2a-416f-9726-ffa469aecdb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2cf142b1ede649e1bfdf05a6b3196e0a'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:50.577554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1181/2371] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:50 2022] DELETE /identity/v3/users/2cf142b1ede649e1bfdf05a6b3196e0a => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:50.964605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:50.967117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:50.996367 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9ec54b-2ff2-4110-b38d-a82b2eec6919 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:51.093439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:51.094559 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.094978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.095482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] PATH_INFO: `/v3/projects/7099aeed78114a77b08c019d4bb01a78` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.097019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7099aeed78114a77b08c019d4bb01a78)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:51.100719 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:51.162117 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:51.186653 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7099aeed78114a77b08c019d4bb01a78', 'request_id': 'req-7a57c001-82e7-4394-91ec-5590a206dcc6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:51.194055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7099aeed78114a77b08c019d4bb01a78', 'request_id': 'req-7a57c001-82e7-4394-91ec-5590a206dcc6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:51.203093 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7099aeed78114a77b08c019d4bb01a78', 'request_id': 'req-7a57c001-82e7-4394-91ec-5590a206dcc6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:51.217266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] The token cache is being invalidate because project 7099aeed78114a77b08c019d4bb01a78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:51.217557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:51.218242 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7a57c001-82e7-4394-91ec-5590a206dcc6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:51.220861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1191/2372] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:50 2022] DELETE /identity/v3/projects/7099aeed78114a77b08c019d4bb01a78 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:51.227507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b1abb-90c3-41b4-a8f9-9eb3a4a955c0 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.227844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b1abb-90c3-41b4-a8f9-9eb3a4a955c0 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.228161 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b1abb-90c3-41b4-a8f9-9eb3a4a955c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.256610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d80b1abb-90c3-41b4-a8f9-9eb3a4a955c0 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:51.258950 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d80b1abb-90c3-41b4-a8f9-9eb3a4a955c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:51.359780 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1182/2373] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:51.370089 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b26c15-16f5-40ef-ad8d-945366318ddd None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.370728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b26c15-16f5-40ef-ad8d-945366318ddd None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.370728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b26c15-16f5-40ef-ad8d-945366318ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.394206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-50b26c15-16f5-40ef-ad8d-945366318ddd None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:51.395616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-50b26c15-16f5-40ef-ad8d-945366318ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:51.404387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1192/2374] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:51.409919 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b1abb-90c3-41b4-a8f9-9eb3a4a955c0 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:51.420990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b1abb-90c3-41b4-a8f9-9eb3a4a955c0 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:51.427071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:51.427723 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.427999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.428273 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.428952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:51.429242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:51.430964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:51.444422 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1183/2375] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:48:51.449968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b26c15-16f5-40ef-ad8d-945366318ddd None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:51.461851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b26c15-16f5-40ef-ad8d-945366318ddd None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:51.468743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a62981-01e1-4ca8-aa56-fae6b1ab9016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:51.469668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a62981-01e1-4ca8-aa56-fae6b1ab9016 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.470950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a62981-01e1-4ca8-aa56-fae6b1ab9016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.470950 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a62981-01e1-4ca8-aa56-fae6b1ab9016 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.471365 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a62981-01e1-4ca8-aa56-fae6b1ab9016 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:51.473366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a62981-01e1-4ca8-aa56-fae6b1ab9016 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:51.507504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1193/2376] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:51 2022] POST /identity/v3/projects => generated 377 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:51.511945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:51.523938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a013371e-2ce0-4313-b965-27be77c33e20 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:51.548412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:51.603044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23df607-2510-4e2f-a144-15cf7f47dfbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:51.603666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23df607-2510-4e2f-a144-15cf7f47dfbe admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.603946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23df607-2510-4e2f-a144-15cf7f47dfbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.604216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23df607-2510-4e2f-a144-15cf7f47dfbe admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.604867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23df607-2510-4e2f-a144-15cf7f47dfbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:51.606865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23df607-2510-4e2f-a144-15cf7f47dfbe admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:51.630264 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1184/2377] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:51 2022] POST /identity/v3/users => generated 289 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:51.636820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a62981-01e1-4ca8-aa56-fae6b1ab9016 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:51.656318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a62981-01e1-4ca8-aa56-fae6b1ab9016 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:51.664809 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8d6f9-7e72-4f22-bbc9-4d1cd5f6143f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:51.665884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8d6f9-7e72-4f22-bbc9-4d1cd5f6143f admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.666391 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8d6f9-7e72-4f22-bbc9-4d1cd5f6143f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.666915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8d6f9-7e72-4f22-bbc9-4d1cd5f6143f admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.668058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf8d6f9-7e72-4f22-bbc9-4d1cd5f6143f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:51.671294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf8d6f9-7e72-4f22-bbc9-4d1cd5f6143f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:51.687743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1194/2378] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:51 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:51.694173 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23df607-2510-4e2f-a144-15cf7f47dfbe admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:51.707121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23df607-2510-4e2f-a144-15cf7f47dfbe admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:51.712434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4e463c-fc44-4f52-aac1-0b45964bdd08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:51.713053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4e463c-fc44-4f52-aac1-0b45964bdd08 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.713330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4e463c-fc44-4f52-aac1-0b45964bdd08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.713608 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4e463c-fc44-4f52-aac1-0b45964bdd08 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.714291 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4e463c-fc44-4f52-aac1-0b45964bdd08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:51.715994 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4e463c-fc44-4f52-aac1-0b45964bdd08 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:51.728731 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1185/2379] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:51 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:51.733270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8d6f9-7e72-4f22-bbc9-4d1cd5f6143f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:51.744669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8d6f9-7e72-4f22-bbc9-4d1cd5f6143f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:51.749754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:51.750534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.751728 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.752048 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] PATH_INFO: `/v3/projects/382d055dde074f4c95169a4d671b8a56/users/8f6f7641edb14917878a55546a70d5d3/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.771680 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] RBAC: Authorizing `identity:create_grant(project_id=382d055dde074f4c95169a4d671b8a56, user_id=8f6f7641edb14917878a55546a70d5d3, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:51.775121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:51.791371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1195/2380] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:51 2022] PUT /identity/v3/projects/382d055dde074f4c95169a4d671b8a56/users/8f6f7641edb14917878a55546a70d5d3/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:51.803300 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfcb364-0445-4ffe-8872-dc3f61f4f298 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.803722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfcb364-0445-4ffe-8872-dc3f61f4f298 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.804096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfcb364-0445-4ffe-8872-dc3f61f4f298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:51.837296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-adfcb364-0445-4ffe-8872-dc3f61f4f298 None None] MFA Rules not processed for user `8f6f7641edb14917878a55546a70d5d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:51.838582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-adfcb364-0445-4ffe-8872-dc3f61f4f298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:51.912898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1186/2381] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:51 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:51.926516 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:51.928332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:51.992784 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a3dece-ace3-47bd-baed-e8a7ebe3b62f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:51.996557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb9b963-47b7-4640-b812-52f5496f5f86 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e3f6be584e34bfb9e430c7756d071d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:51.997181 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb9b963-47b7-4640-b812-52f5496f5f86 service glance] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:51.997463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb9b963-47b7-4640-b812-52f5496f5f86 service glance] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:51.997735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb9b963-47b7-4640-b812-52f5496f5f86 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:52.008502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb9b963-47b7-4640-b812-52f5496f5f86 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:52.010147 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb9b963-47b7-4640-b812-52f5496f5f86 service glance] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:52.021754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1196/2382] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:51 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:54.551027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfcb364-0445-4ffe-8872-dc3f61f4f298 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:54.553058 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-adfcb364-0445-4ffe-8872-dc3f61f4f298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:54.669261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfcb364-0445-4ffe-8872-dc3f61f4f298 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:54.674753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:54.675428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:54.675706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:54.675982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] PATH_INFO: `/v3/users/2ad62e7a918e47848604ef80303d01cc` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:54.677309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ad62e7a918e47848604ef80303d01cc)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:54.679413 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:54.729064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:54.747852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ad62e7a918e47848604ef80303d01cc'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:54.756851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ad62e7a918e47848604ef80303d01cc'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:54.757657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ad62e7a918e47848604ef80303d01cc'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:54.758359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ad62e7a918e47848604ef80303d01cc'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:54.768307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ad62e7a918e47848604ef80303d01cc'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:54.790198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1187/2383] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:48:54 2022] DELETE /identity/v3/users/2ad62e7a918e47848604ef80303d01cc => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:54.795836 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db16e21-ed6d-4f5d-926d-87bf5e47e830 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:54.796654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db16e21-ed6d-4f5d-926d-87bf5e47e830 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:54.796654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db16e21-ed6d-4f5d-926d-87bf5e47e830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:54.817301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0db16e21-ed6d-4f5d-926d-87bf5e47e830 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:54.818587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0db16e21-ed6d-4f5d-926d-87bf5e47e830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:54.879503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:54.892687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1197/2384] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:54.905073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:54.906598 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:54.977170 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d575c769-9111-496d-bdd4-ed80409ee2d2 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:54.981371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422bd876-c08c-4594-89b6-fe6f2111526c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:54.982002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422bd876-c08c-4594-89b6-fe6f2111526c service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:54.982279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422bd876-c08c-4594-89b6-fe6f2111526c service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:54.982593 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422bd876-c08c-4594-89b6-fe6f2111526c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:54.994329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422bd876-c08c-4594-89b6-fe6f2111526c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:54.995843 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422bd876-c08c-4594-89b6-fe6f2111526c service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:55.007532 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1188/2385] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:55.367850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db16e21-ed6d-4f5d-926d-87bf5e47e830 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:55.371328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0db16e21-ed6d-4f5d-926d-87bf5e47e830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:55.400828 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db16e21-ed6d-4f5d-926d-87bf5e47e830 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:55.409427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:55.410696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:55.411298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:55.411798 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] PATH_INFO: `/v3/projects/81eba63c7ea8490d8a9a6600449267f4` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:55.414439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81eba63c7ea8490d8a9a6600449267f4)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:55.418989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:55.487765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81eba63c7ea8490d8a9a6600449267f4', 'request_id': 'req-4b695ce9-42cd-47b9-89f3-26f9a0a19356'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:55.495417 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81eba63c7ea8490d8a9a6600449267f4', 'request_id': 'req-4b695ce9-42cd-47b9-89f3-26f9a0a19356'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:55.505886 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81eba63c7ea8490d8a9a6600449267f4', 'request_id': 'req-4b695ce9-42cd-47b9-89f3-26f9a0a19356'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:55.520020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] The token cache is being invalidate because project 81eba63c7ea8490d8a9a6600449267f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:48:55.520440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:55.521234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-4b695ce9-42cd-47b9-89f3-26f9a0a19356 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:48:55.523027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1198/2386] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:48:55 2022] DELETE /identity/v3/projects/81eba63c7ea8490d8a9a6600449267f4 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:48:55.528359 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1cce5-766c-47ef-a564-9d83adbd6043 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:55.528750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1cce5-766c-47ef-a564-9d83adbd6043 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:55.529024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1cce5-766c-47ef-a564-9d83adbd6043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:55.552657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-a4c1cce5-766c-47ef-a564-9d83adbd6043 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:55.553899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a4c1cce5-766c-47ef-a564-9d83adbd6043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:55.629122 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1189/2387] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:48:55.638389 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12043852-e81b-496f-b71e-c5d60dbe45f9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:55.638703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12043852-e81b-496f-b71e-c5d60dbe45f9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:55.638998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12043852-e81b-496f-b71e-c5d60dbe45f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:55.662377 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-12043852-e81b-496f-b71e-c5d60dbe45f9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:55.667328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-12043852-e81b-496f-b71e-c5d60dbe45f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:55.673639 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1199/2388] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:55.678231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c1cce5-766c-47ef-a564-9d83adbd6043 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:55.693301 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c1cce5-766c-47ef-a564-9d83adbd6043 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:55.699412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:55.700357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:55.700638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:55.701029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:55.701836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:48:55.702220 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:55.704596 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:55.718849 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1190/2389] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:48:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:48:55.725651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12043852-e81b-496f-b71e-c5d60dbe45f9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:55.737753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12043852-e81b-496f-b71e-c5d60dbe45f9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:55.743294 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb231883-5398-4bdf-a2d7-f38c2df044ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:55.743920 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb231883-5398-4bdf-a2d7-f38c2df044ad admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:55.744201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb231883-5398-4bdf-a2d7-f38c2df044ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:55.744473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb231883-5398-4bdf-a2d7-f38c2df044ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:55.745107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb231883-5398-4bdf-a2d7-f38c2df044ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:55.747081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb231883-5398-4bdf-a2d7-f38c2df044ad admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:55.785754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1200/2390] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:48:55 2022] POST /identity/v3/projects => generated 373 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:48:55.787973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:55.803708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f47f8a-905f-4cc5-a83c-f58242f55403 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:55.868536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:48:55.891823 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdda3a9-ec4a-4e94-af83-0643f59548b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:55.892498 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdda3a9-ec4a-4e94-af83-0643f59548b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:55.892783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdda3a9-ec4a-4e94-af83-0643f59548b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:55.893106 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdda3a9-ec4a-4e94-af83-0643f59548b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:55.893761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fdda3a9-ec4a-4e94-af83-0643f59548b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:55.895997 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fdda3a9-ec4a-4e94-af83-0643f59548b0 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:55.923647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1191/2391] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:48:55 2022] POST /identity/v3/users => generated 287 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:48:55.927742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb231883-5398-4bdf-a2d7-f38c2df044ad admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:55.939132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb231883-5398-4bdf-a2d7-f38c2df044ad admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:55.945503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceafad5c-8240-4da7-893f-677c63112de4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:55.946130 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceafad5c-8240-4da7-893f-677c63112de4 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:55.946405 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceafad5c-8240-4da7-893f-677c63112de4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:55.946714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceafad5c-8240-4da7-893f-677c63112de4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:55.947394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceafad5c-8240-4da7-893f-677c63112de4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:55.949020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceafad5c-8240-4da7-893f-677c63112de4 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:55.961439 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1201/2392] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:55 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:55.967198 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdda3a9-ec4a-4e94-af83-0643f59548b0 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:55.978808 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdda3a9-ec4a-4e94-af83-0643f59548b0 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:55.983866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d95897a-9a47-4c0b-839a-db0cf9d388ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:55.984483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95897a-9a47-4c0b-839a-db0cf9d388ed admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:55.984758 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95897a-9a47-4c0b-839a-db0cf9d388ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:55.985137 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95897a-9a47-4c0b-839a-db0cf9d388ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:55.985865 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d95897a-9a47-4c0b-839a-db0cf9d388ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:55.987508 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d95897a-9a47-4c0b-839a-db0cf9d388ed admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:55.999452 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1192/2393] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:48:55 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:48:56.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceafad5c-8240-4da7-893f-677c63112de4 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:56.016004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceafad5c-8240-4da7-893f-677c63112de4 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:56.021671 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:56.022343 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:56.022651 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:56.023018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] PATH_INFO: `/v3/projects/2a1efe09c74b46f39c467c6a602abd25/users/796ecf06911540789ec035bc41d52031/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:56.040944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a1efe09c74b46f39c467c6a602abd25, user_id=796ecf06911540789ec035bc41d52031, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:56.044026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:56.062688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1202/2394] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:48:56 2022] PUT /identity/v3/projects/2a1efe09c74b46f39c467c6a602abd25/users/796ecf06911540789ec035bc41d52031/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:48:56.074644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d16a90b-c053-451d-a3f1-c1b63c074759 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:56.075006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d16a90b-c053-451d-a3f1-c1b63c074759 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:56.075197 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d16a90b-c053-451d-a3f1-c1b63c074759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:56.113362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-7d16a90b-c053-451d-a3f1-c1b63c074759 None None] MFA Rules not processed for user `796ecf06911540789ec035bc41d52031`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:56.114805 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7d16a90b-c053-451d-a3f1-c1b63c074759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:56.195814 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1193/2395] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:48:56.207803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:48:56.210151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:56.305413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338f7e4e-9737-4f3d-a7d5-292d3854c16e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:48:56.309742 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:48:56.310407 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:56.310735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:56.311034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:56.321143 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:48:56.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:48:56.335793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1203/2396] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:48:56 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:48:57.777749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2d9bc9-1dc3-4889-9192-7254294b21fc None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:48:57.778365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2d9bc9-1dc3-4889-9192-7254294b21fc None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:48:57.778365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2d9bc9-1dc3-4889-9192-7254294b21fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:48:57.801284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8d2d9bc9-1dc3-4889-9192-7254294b21fc None None] MFA Rules not processed for user `796ecf06911540789ec035bc41d52031`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:48:57.802644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8d2d9bc9-1dc3-4889-9192-7254294b21fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:48:57.809989 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1194/2397] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:48:57 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:06.329418 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:07.793709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=51,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:09.214261 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:09.215760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:09.292074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d3cf3a-41e6-49b5-96e7-2c709c3a9e31 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:09.296989 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:09.297676 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:09.297959 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:09.298267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] PATH_INFO: `/v3/users/8f6f7641edb14917878a55546a70d5d3` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:09.299691 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f6f7641edb14917878a55546a70d5d3)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:09.301924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:09.360020 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f6f7641edb14917878a55546a70d5d3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:09.368767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f6f7641edb14917878a55546a70d5d3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:09.370538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f6f7641edb14917878a55546a70d5d3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:09.371826 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f6f7641edb14917878a55546a70d5d3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:09.381247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f6f7641edb14917878a55546a70d5d3'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:09.405859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1204/2398] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:49:09 2022] DELETE /identity/v3/users/8f6f7641edb14917878a55546a70d5d3 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:49:09.412584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a76628a-50de-47c8-a2cb-a216af0deac0 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:09.413436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a76628a-50de-47c8-a2cb-a216af0deac0 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:09.413634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a76628a-50de-47c8-a2cb-a216af0deac0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:09.440749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6a76628a-50de-47c8-a2cb-a216af0deac0 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:09.443042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6a76628a-50de-47c8-a2cb-a216af0deac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:09.543630 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1195/2399] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:09.554360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:09.556765 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:09.649360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f129e389-8afa-45a5-8b2a-60befe2468ec admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:09.653440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06ac541-fbbb-4cbe-b791-868347cdecf6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:09.654069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06ac541-fbbb-4cbe-b791-868347cdecf6 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:09.654350 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06ac541-fbbb-4cbe-b791-868347cdecf6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:09.654716 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06ac541-fbbb-4cbe-b791-868347cdecf6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:09.666592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06ac541-fbbb-4cbe-b791-868347cdecf6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:09.668590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06ac541-fbbb-4cbe-b791-868347cdecf6 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:09.681223 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1205/2400] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:49:10.002456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a76628a-50de-47c8-a2cb-a216af0deac0 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:10.004458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6a76628a-50de-47c8-a2cb-a216af0deac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:10.022820 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a76628a-50de-47c8-a2cb-a216af0deac0 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:10.028784 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:10.029482 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.029765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.030037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] PATH_INFO: `/v3/projects/382d055dde074f4c95169a4d671b8a56` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.031284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] RBAC: Authorizing `identity:delete_project(project_id=382d055dde074f4c95169a4d671b8a56)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:10.035324 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:10.092679 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '382d055dde074f4c95169a4d671b8a56', 'request_id': 'req-48ae84be-ed09-46cf-a44b-8069aaecda0d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:10.100590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '382d055dde074f4c95169a4d671b8a56', 'request_id': 'req-48ae84be-ed09-46cf-a44b-8069aaecda0d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:10.107889 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '382d055dde074f4c95169a4d671b8a56', 'request_id': 'req-48ae84be-ed09-46cf-a44b-8069aaecda0d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:10.123025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] The token cache is being invalidate because project 382d055dde074f4c95169a4d671b8a56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:49:10.123311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:10.124036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-48ae84be-ed09-46cf-a44b-8069aaecda0d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:10.125899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1196/2401] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:49:10 2022] DELETE /identity/v3/projects/382d055dde074f4c95169a4d671b8a56 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:49:10.131442 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8419e16-2189-40f7-9abf-6fbdd8a8d015 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.131740 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8419e16-2189-40f7-9abf-6fbdd8a8d015 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.132026 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8419e16-2189-40f7-9abf-6fbdd8a8d015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.153957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-b8419e16-2189-40f7-9abf-6fbdd8a8d015 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:10.155263 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-b8419e16-2189-40f7-9abf-6fbdd8a8d015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:10.176060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:10.228687 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1206/2402] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:10.237383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c66e9cf-4480-467c-bb35-43348609083d None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.237707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c66e9cf-4480-467c-bb35-43348609083d None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.237913 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c66e9cf-4480-467c-bb35-43348609083d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.274326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2c66e9cf-4480-467c-bb35-43348609083d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:10.275640 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2c66e9cf-4480-467c-bb35-43348609083d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:10.286169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1197/2403] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:10.290897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8419e16-2189-40f7-9abf-6fbdd8a8d015 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:10.304749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8419e16-2189-40f7-9abf-6fbdd8a8d015 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:10.310599 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:10.311238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.311524 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.311803 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.312477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:49:10.313396 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:10.315189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:10.333176 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1207/2404] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:49:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:49:10.338171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c66e9cf-4480-467c-bb35-43348609083d None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:10.348883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c66e9cf-4480-467c-bb35-43348609083d None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:10.354076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d3f684-1499-4770-9d25-3a7e4ea76045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:10.354749 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d3f684-1499-4770-9d25-3a7e4ea76045 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.355051 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d3f684-1499-4770-9d25-3a7e4ea76045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.355330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d3f684-1499-4770-9d25-3a7e4ea76045 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.355992 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d3f684-1499-4770-9d25-3a7e4ea76045 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:10.357983 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d3f684-1499-4770-9d25-3a7e4ea76045 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:10.394108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1198/2405] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:49:10 2022] POST /identity/v3/projects => generated 359 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:49:10.399005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:10.411194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4365a841-f02f-45df-9647-19d5bf70b2fe admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:10.483892 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bba0a7-2641-4406-a05a-4c3a936ef0cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:10.484527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bba0a7-2641-4406-a05a-4c3a936ef0cd admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.484816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bba0a7-2641-4406-a05a-4c3a936ef0cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.485090 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bba0a7-2641-4406-a05a-4c3a936ef0cd admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.485758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bba0a7-2641-4406-a05a-4c3a936ef0cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:10.487774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bba0a7-2641-4406-a05a-4c3a936ef0cd admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:10.513542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1208/2406] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:49:10 2022] POST /identity/v3/users => generated 280 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:49:10.520733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d3f684-1499-4770-9d25-3a7e4ea76045 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:10.533537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d3f684-1499-4770-9d25-3a7e4ea76045 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:10.538666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba21335-7b25-424e-a19a-ae53354e042c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:10.539301 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba21335-7b25-424e-a19a-ae53354e042c admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.539583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba21335-7b25-424e-a19a-ae53354e042c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.539857 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba21335-7b25-424e-a19a-ae53354e042c admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.540522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba21335-7b25-424e-a19a-ae53354e042c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:10.542181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba21335-7b25-424e-a19a-ae53354e042c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:10.560414 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1199/2407] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:10 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:49:10.566404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bba0a7-2641-4406-a05a-4c3a936ef0cd admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:10.580759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bba0a7-2641-4406-a05a-4c3a936ef0cd admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:10.587322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d7d91f-5d74-483b-a5f6-5438eaf136d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:10.587963 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d7d91f-5d74-483b-a5f6-5438eaf136d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.588239 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d7d91f-5d74-483b-a5f6-5438eaf136d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.588561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d7d91f-5d74-483b-a5f6-5438eaf136d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.589217 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d7d91f-5d74-483b-a5f6-5438eaf136d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:10.590880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d7d91f-5d74-483b-a5f6-5438eaf136d2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:10.603807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1209/2408] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:10 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:49:10.608336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba21335-7b25-424e-a19a-ae53354e042c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:10.620036 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba21335-7b25-424e-a19a-ae53354e042c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:10.624945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:10.625970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.626104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.626311 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] PATH_INFO: `/v3/projects/bc3ce3ba99a549a481ae27a1338b5f18/users/39d86906a60c43f8a2ea9de0055fd024/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.643899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc3ce3ba99a549a481ae27a1338b5f18, user_id=39d86906a60c43f8a2ea9de0055fd024, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:10.646841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:10.663662 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1200/2409] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:49:10 2022] PUT /identity/v3/projects/bc3ce3ba99a549a481ae27a1338b5f18/users/39d86906a60c43f8a2ea9de0055fd024/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:10.676197 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7575ad-934b-4a22-a869-2cc305a20383 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.676565 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7575ad-934b-4a22-a869-2cc305a20383 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.676990 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7575ad-934b-4a22-a869-2cc305a20383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.713616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ba7575ad-934b-4a22-a869-2cc305a20383 None None] MFA Rules not processed for user `39d86906a60c43f8a2ea9de0055fd024`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:10.715752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ba7575ad-934b-4a22-a869-2cc305a20383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:10.819859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1210/2410] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:10 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:10.870886 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:10.873336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:10.907331 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:10.967211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3998fccd-3cd9-43ef-9eca-4676c68d51c8 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:10.975904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6372d28c-3487-4066-8731-68281e71ea49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:10.976892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6372d28c-3487-4066-8731-68281e71ea49 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:10.977352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6372d28c-3487-4066-8731-68281e71ea49 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:10.977844 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6372d28c-3487-4066-8731-68281e71ea49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:10.992837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6372d28c-3487-4066-8731-68281e71ea49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:10.995054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6372d28c-3487-4066-8731-68281e71ea49 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:11.011536 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1201/2411] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:10 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:49:11.034298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7575ad-934b-4a22-a869-2cc305a20383 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:11.067320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:11.124218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7575ad-934b-4a22-a869-2cc305a20383 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:11.131481 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:11.132578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:11.133060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:11.133534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] PATH_INFO: `/v3/users/39d86906a60c43f8a2ea9de0055fd024` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:11.135735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39d86906a60c43f8a2ea9de0055fd024)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:11.139701 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:11.233164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39d86906a60c43f8a2ea9de0055fd024'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:11.248975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39d86906a60c43f8a2ea9de0055fd024'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:11.251183 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39d86906a60c43f8a2ea9de0055fd024'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:11.252208 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39d86906a60c43f8a2ea9de0055fd024'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:11.265107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39d86906a60c43f8a2ea9de0055fd024'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:11.301403 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1211/2412] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:49:11 2022] DELETE /identity/v3/users/39d86906a60c43f8a2ea9de0055fd024 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:11.308431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd7459-5dd3-4829-8337-7be44f5c2534 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:11.308940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd7459-5dd3-4829-8337-7be44f5c2534 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:11.309436 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd7459-5dd3-4829-8337-7be44f5c2534 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:11.339924 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-d8fd7459-5dd3-4829-8337-7be44f5c2534 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:11.341799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d8fd7459-5dd3-4829-8337-7be44f5c2534 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:11.460402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1202/2413] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:11.473575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:11.475941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:11.560502 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84463edc-8bf5-45e5-b1a8-ab9d799d3631 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:11.565695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6534ef6-e914-4969-9a51-689e1d2cb884 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:11.566446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6534ef6-e914-4969-9a51-689e1d2cb884 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:11.566863 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6534ef6-e914-4969-9a51-689e1d2cb884 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:11.567255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6534ef6-e914-4969-9a51-689e1d2cb884 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:11.579032 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6534ef6-e914-4969-9a51-689e1d2cb884 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:11.580753 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6534ef6-e914-4969-9a51-689e1d2cb884 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:11.591552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1212/2414] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:49:11.917794 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd7459-5dd3-4829-8337-7be44f5c2534 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:11.920254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-d8fd7459-5dd3-4829-8337-7be44f5c2534 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:11.944928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd7459-5dd3-4829-8337-7be44f5c2534 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:11.951617 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:11.952435 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:11.952895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:11.953233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] PATH_INFO: `/v3/projects/bc3ce3ba99a549a481ae27a1338b5f18` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:11.955850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc3ce3ba99a549a481ae27a1338b5f18)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:11.957815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:12.014475 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc3ce3ba99a549a481ae27a1338b5f18', 'request_id': 'req-98739b79-39c4-485b-868b-b980ec62871c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:12.022335 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc3ce3ba99a549a481ae27a1338b5f18', 'request_id': 'req-98739b79-39c4-485b-868b-b980ec62871c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:12.029509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc3ce3ba99a549a481ae27a1338b5f18', 'request_id': 'req-98739b79-39c4-485b-868b-b980ec62871c'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:12.044416 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] The token cache is being invalidate because project bc3ce3ba99a549a481ae27a1338b5f18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:49:12.044715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:12.045376 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-98739b79-39c4-485b-868b-b980ec62871c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:12.047181 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1203/2415] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:49:11 2022] DELETE /identity/v3/projects/bc3ce3ba99a549a481ae27a1338b5f18 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:12.052578 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d832684-417d-4346-9ed4-5f1180aedc71 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.052902 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d832684-417d-4346-9ed4-5f1180aedc71 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.053217 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d832684-417d-4346-9ed4-5f1180aedc71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.075528 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-8d832684-417d-4346-9ed4-5f1180aedc71 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:12.076907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-8d832684-417d-4346-9ed4-5f1180aedc71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:12.131957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:12.181359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1213/2416] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:12.191400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2cfd6d-6128-4c8a-8026-8ac880043902 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.191830 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2cfd6d-6128-4c8a-8026-8ac880043902 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.192256 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2cfd6d-6128-4c8a-8026-8ac880043902 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.215451 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-3f2cfd6d-6128-4c8a-8026-8ac880043902 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:12.216708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3f2cfd6d-6128-4c8a-8026-8ac880043902 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:12.226107 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1204/2417] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:12.230974 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d832684-417d-4346-9ed4-5f1180aedc71 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:12.244789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d832684-417d-4346-9ed4-5f1180aedc71 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:12.251185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:12.252045 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.252393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.252867 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.253422 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:49:12.253868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:12.255666 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:12.278699 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1214/2418] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:49:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:49:12.279965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2cfd6d-6128-4c8a-8026-8ac880043902 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:12.291641 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2cfd6d-6128-4c8a-8026-8ac880043902 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:12.296628 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d24f6b-aa2b-4646-932b-97b28f32d831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:12.298355 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d24f6b-aa2b-4646-932b-97b28f32d831 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.298670 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d24f6b-aa2b-4646-932b-97b28f32d831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.298945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d24f6b-aa2b-4646-932b-97b28f32d831 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.299612 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d24f6b-aa2b-4646-932b-97b28f32d831 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:12.301554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d24f6b-aa2b-4646-932b-97b28f32d831 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:12.335603 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1205/2419] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:49:12 2022] POST /identity/v3/projects => generated 397 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:49:12.340341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:12.352885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cddd2e4-1725-4502-8dab-8a63c491c329 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:12.454906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70602e95-6b9c-4bfb-843f-154a6b8f8a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:12.455630 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70602e95-6b9c-4bfb-843f-154a6b8f8a88 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.456057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70602e95-6b9c-4bfb-843f-154a6b8f8a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.456416 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70602e95-6b9c-4bfb-843f-154a6b8f8a88 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.457577 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70602e95-6b9c-4bfb-843f-154a6b8f8a88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:12.460556 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70602e95-6b9c-4bfb-843f-154a6b8f8a88 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:12.505648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1215/2420] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:49:12 2022] POST /identity/v3/users => generated 299 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:49:12.511046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d24f6b-aa2b-4646-932b-97b28f32d831 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:12.529766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d24f6b-aa2b-4646-932b-97b28f32d831 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:12.541193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f15008-ac6a-4528-badf-018513807ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:12.542029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f15008-ac6a-4528-badf-018513807ad4 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.542364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f15008-ac6a-4528-badf-018513807ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.542915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f15008-ac6a-4528-badf-018513807ad4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.543709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f15008-ac6a-4528-badf-018513807ad4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:12.545769 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f15008-ac6a-4528-badf-018513807ad4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:12.569202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1206/2421] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:12 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:49:12.574580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70602e95-6b9c-4bfb-843f-154a6b8f8a88 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:12.590985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70602e95-6b9c-4bfb-843f-154a6b8f8a88 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:12.598388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05161ae-c65f-4792-9218-cc3c044bcf06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:12.599291 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05161ae-c65f-4792-9218-cc3c044bcf06 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.599623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05161ae-c65f-4792-9218-cc3c044bcf06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.600114 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05161ae-c65f-4792-9218-cc3c044bcf06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.601504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05161ae-c65f-4792-9218-cc3c044bcf06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:12.603426 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05161ae-c65f-4792-9218-cc3c044bcf06 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:12.617238 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1216/2422] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:12 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:49:12.621697 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f15008-ac6a-4528-badf-018513807ad4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:12.635965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f15008-ac6a-4528-badf-018513807ad4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:12.645000 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:12.646126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.646681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.647148 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] PATH_INFO: `/v3/projects/65417f6732ff4df4b3e69cc83845f080/users/7d1a27272f5c43298bbd8c5c29d71dab/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.651134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:12.677129 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] RBAC: Authorizing `identity:create_grant(project_id=65417f6732ff4df4b3e69cc83845f080, user_id=7d1a27272f5c43298bbd8c5c29d71dab, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:12.682917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:12.707257 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1207/2423] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:49:12 2022] PUT /identity/v3/projects/65417f6732ff4df4b3e69cc83845f080/users/7d1a27272f5c43298bbd8c5c29d71dab/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:12.718292 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881c573c-2bff-4b77-8753-3212f76572ff None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.718612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881c573c-2bff-4b77-8753-3212f76572ff None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.719194 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881c573c-2bff-4b77-8753-3212f76572ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.752769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-881c573c-2bff-4b77-8753-3212f76572ff None None] MFA Rules not processed for user `7d1a27272f5c43298bbd8c5c29d71dab`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:12.754008 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-881c573c-2bff-4b77-8753-3212f76572ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:12.848600 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1217/2424] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:12 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:12.863822 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:12.865696 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:12.942179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251d970-0bb2-4da9-94d3-1286e2422fae admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:12.947607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:12.948296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:12.948610 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:12.948892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:12.958980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:12.960449 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:12.971174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1208/2425] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:12 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:49:15.041859 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb2c3ce-be9e-4077-b3c8-1620591de6bb None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:15.042457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb2c3ce-be9e-4077-b3c8-1620591de6bb None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:15.042894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb2c3ce-be9e-4077-b3c8-1620591de6bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:15.079847 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0bb2c3ce-be9e-4077-b3c8-1620591de6bb None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:15.081011 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0bb2c3ce-be9e-4077-b3c8-1620591de6bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:15.105050 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:15.156039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1218/2426] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:49:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:15.218054 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:15.220860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:15.307683 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ab9b1e-a3aa-4e3a-ab20-5f5c577eaf50 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:15.313715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ca4285-6f55-48dc-a5f1-ac036d6d8ff1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:15.314529 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ca4285-6f55-48dc-a5f1-ac036d6d8ff1 service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:15.314894 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ca4285-6f55-48dc-a5f1-ac036d6d8ff1 service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:15.315268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ca4285-6f55-48dc-a5f1-ac036d6d8ff1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:15.325056 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ca4285-6f55-48dc-a5f1-ac036d6d8ff1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:15.326702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ca4285-6f55-48dc-a5f1-ac036d6d8ff1 service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:15.336757 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1209/2427] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:49:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:49:15.916173 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50cd067-d6bd-4cf7-b6c7-dea266a156d2 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:15.916465 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50cd067-d6bd-4cf7-b6c7-dea266a156d2 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:15.916739 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50cd067-d6bd-4cf7-b6c7-dea266a156d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:15.939488 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-c50cd067-d6bd-4cf7-b6c7-dea266a156d2 None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:15.940743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c50cd067-d6bd-4cf7-b6c7-dea266a156d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:15.948751 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1219/2428] 10.222.0.26 () {58 vars in 967 bytes} [Wed Oct 19 00:49:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:15.963265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ca4285-6f55-48dc-a5f1-ac036d6d8ff1 service swift] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:15.975440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ca4285-6f55-48dc-a5f1-ac036d6d8ff1 service swift] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:15.980777 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a997f361-7178-4f07-95fe-b40a10b2592d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:15.981389 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a997f361-7178-4f07-95fe-b40a10b2592d service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:15.981671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a997f361-7178-4f07-95fe-b40a10b2592d service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:15.981955 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a997f361-7178-4f07-95fe-b40a10b2592d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:15.990999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a997f361-7178-4f07-95fe-b40a10b2592d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:15.992605 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a997f361-7178-4f07-95fe-b40a10b2592d service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:16.008521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1210/2429] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:49:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:49:16.340402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d6157-9aa7-4fed-b91f-92a06c214b19 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:16.341267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d6157-9aa7-4fed-b91f-92a06c214b19 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:16.341267 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d6157-9aa7-4fed-b91f-92a06c214b19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:16.365634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-738d6157-9aa7-4fed-b91f-92a06c214b19 None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:16.366934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-738d6157-9aa7-4fed-b91f-92a06c214b19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:16.376395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1220/2430] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:49:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:16.390959 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a997f361-7178-4f07-95fe-b40a10b2592d service swift] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:16.403433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a997f361-7178-4f07-95fe-b40a10b2592d service swift] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:16.409768 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418dff12-7b8c-482b-b090-73556f33e0d4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:16.410396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418dff12-7b8c-482b-b090-73556f33e0d4 service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:16.410712 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418dff12-7b8c-482b-b090-73556f33e0d4 service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:16.410987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418dff12-7b8c-482b-b090-73556f33e0d4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:16.420191 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418dff12-7b8c-482b-b090-73556f33e0d4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:16.422765 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418dff12-7b8c-482b-b090-73556f33e0d4 service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:16.432815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1211/2431] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:49:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:49:16.835322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac1ed39-12a9-47a3-ad7e-d1a2620c239a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:16.835806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac1ed39-12a9-47a3-ad7e-d1a2620c239a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:16.836218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac1ed39-12a9-47a3-ad7e-d1a2620c239a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:16.858923 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-aac1ed39-12a9-47a3-ad7e-d1a2620c239a None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:16.860778 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-aac1ed39-12a9-47a3-ad7e-d1a2620c239a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:16.868713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1221/2432] 10.222.0.26 () {58 vars in 967 bytes} [Wed Oct 19 00:49:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:16.881904 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418dff12-7b8c-482b-b090-73556f33e0d4 service swift] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:16.895244 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418dff12-7b8c-482b-b090-73556f33e0d4 service swift] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:16.900868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100c5f73-0b7b-4d27-abe1-8f9c1a100c3d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:16.901489 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100c5f73-0b7b-4d27-abe1-8f9c1a100c3d service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:16.901772 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100c5f73-0b7b-4d27-abe1-8f9c1a100c3d service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:16.902047 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100c5f73-0b7b-4d27-abe1-8f9c1a100c3d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:16.912016 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100c5f73-0b7b-4d27-abe1-8f9c1a100c3d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:16.913559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100c5f73-0b7b-4d27-abe1-8f9c1a100c3d service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:16.924375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1212/2433] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:49:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:49:19.250060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac1ed39-12a9-47a3-ad7e-d1a2620c239a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:19.252178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-aac1ed39-12a9-47a3-ad7e-d1a2620c239a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:19.352596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac1ed39-12a9-47a3-ad7e-d1a2620c239a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:19.358707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:19.358707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:19.358707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:19.358707 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] PATH_INFO: `/v3/users/796ecf06911540789ec035bc41d52031` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:19.361331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=796ecf06911540789ec035bc41d52031)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:19.363133 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:19.423353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '796ecf06911540789ec035bc41d52031'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:19.432941 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '796ecf06911540789ec035bc41d52031'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:19.433703 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '796ecf06911540789ec035bc41d52031'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:19.434468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '796ecf06911540789ec035bc41d52031'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:19.444948 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '796ecf06911540789ec035bc41d52031'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:19.466697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1222/2434] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:49:19 2022] DELETE /identity/v3/users/796ecf06911540789ec035bc41d52031 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:49:19.472573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9cd215-1a68-4d5c-801d-592bd22a94f1 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:19.473233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9cd215-1a68-4d5c-801d-592bd22a94f1 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:19.473233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9cd215-1a68-4d5c-801d-592bd22a94f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:19.499308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-8b9cd215-1a68-4d5c-801d-592bd22a94f1 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:19.500841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8b9cd215-1a68-4d5c-801d-592bd22a94f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:19.542720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:19.582976 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1213/2435] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:19.596210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:19.598256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:19.631218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:19.665449 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bfc41a-1ce3-4a5b-917b-e3ecfd3b6f6c admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:19.669146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ea8c29-641a-4a43-a3cd-d7da174bfdb9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:19.669793 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ea8c29-641a-4a43-a3cd-d7da174bfdb9 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:19.670069 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ea8c29-641a-4a43-a3cd-d7da174bfdb9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:19.670332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ea8c29-641a-4a43-a3cd-d7da174bfdb9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:19.681216 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ea8c29-641a-4a43-a3cd-d7da174bfdb9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:19.682870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ea8c29-641a-4a43-a3cd-d7da174bfdb9 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:19.693704 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1223/2436] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:49:19.889285 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9cd215-1a68-4d5c-801d-592bd22a94f1 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:19.891669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8b9cd215-1a68-4d5c-801d-592bd22a94f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:19.915936 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9cd215-1a68-4d5c-801d-592bd22a94f1 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:19.926655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:19.927779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:19.928269 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:19.928748 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] PATH_INFO: `/v3/projects/2a1efe09c74b46f39c467c6a602abd25` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:19.932094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a1efe09c74b46f39c467c6a602abd25)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:19.935926 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.003858 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a1efe09c74b46f39c467c6a602abd25', 'request_id': 'req-9369941e-05a4-4e6c-a66e-10cfb09b62f9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.011002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a1efe09c74b46f39c467c6a602abd25', 'request_id': 'req-9369941e-05a4-4e6c-a66e-10cfb09b62f9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.018750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a1efe09c74b46f39c467c6a602abd25', 'request_id': 'req-9369941e-05a4-4e6c-a66e-10cfb09b62f9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.035271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] The token cache is being invalidate because project 2a1efe09c74b46f39c467c6a602abd25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:49:20.035716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.036467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-9369941e-05a4-4e6c-a66e-10cfb09b62f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.038459 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1214/2437] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:49:19 2022] DELETE /identity/v3/projects/2a1efe09c74b46f39c467c6a602abd25 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:20.044087 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1027fa-2c48-4ebf-9dee-01ab00831c8d None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.044463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1027fa-2c48-4ebf-9dee-01ab00831c8d None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.044747 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1027fa-2c48-4ebf-9dee-01ab00831c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.068846 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-0d1027fa-2c48-4ebf-9dee-01ab00831c8d None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:20.070148 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0d1027fa-2c48-4ebf-9dee-01ab00831c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:20.146933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1224/2438] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:20.159397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79385df8-5a85-44c1-b477-6beab3d307e9 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.159699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79385df8-5a85-44c1-b477-6beab3d307e9 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.159953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79385df8-5a85-44c1-b477-6beab3d307e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.183196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-79385df8-5a85-44c1-b477-6beab3d307e9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:20.184455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-79385df8-5a85-44c1-b477-6beab3d307e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:20.193944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1215/2439] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:20.199476 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1027fa-2c48-4ebf-9dee-01ab00831c8d None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.218078 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1027fa-2c48-4ebf-9dee-01ab00831c8d None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.225880 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.226840 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.227165 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.227457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.228353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:49:20.228896 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.232038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.247538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1225/2440] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:49:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:49:20.253817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79385df8-5a85-44c1-b477-6beab3d307e9 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.271374 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79385df8-5a85-44c1-b477-6beab3d307e9 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.276705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.277466 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.277850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.278242 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.279035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.281314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.315387 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1216/2441] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:49:20 2022] POST /identity/v3/projects => generated 389 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:49:20.320290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.332428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54522d42-3873-4d79-8d7f-1b6e358a04af admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.368214 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.370673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:20.424101 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe5cf92-0509-4149-ba6f-eefa0fa5c02e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.424746 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5cf92-0509-4149-ba6f-eefa0fa5c02e admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.425322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5cf92-0509-4149-ba6f-eefa0fa5c02e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.425485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5cf92-0509-4149-ba6f-eefa0fa5c02e admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.426149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe5cf92-0509-4149-ba6f-eefa0fa5c02e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.428164 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe5cf92-0509-4149-ba6f-eefa0fa5c02e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.457503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1226/2442] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:49:20 2022] POST /identity/v3/users => generated 295 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:49:20.462291 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe5cf92-0509-4149-ba6f-eefa0fa5c02e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.477467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca86114-4454-4bac-8b12-3a5ba0f09478 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.482539 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe5cf92-0509-4149-ba6f-eefa0fa5c02e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.483252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.484074 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.484929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.485315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] PATH_INFO: `/v3/users/7d1a27272f5c43298bbd8c5c29d71dab` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.486956 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d1a27272f5c43298bbd8c5c29d71dab)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.489166 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.491433 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a898fb7-f849-4e05-8048-ff7451b1d431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.492537 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a898fb7-f849-4e05-8048-ff7451b1d431 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.493029 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a898fb7-f849-4e05-8048-ff7451b1d431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.493533 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a898fb7-f849-4e05-8048-ff7451b1d431 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.494775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a898fb7-f849-4e05-8048-ff7451b1d431 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.497913 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a898fb7-f849-4e05-8048-ff7451b1d431 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.512155 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1227/2443] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:20 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:49:20.516333 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a898fb7-f849-4e05-8048-ff7451b1d431 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.530417 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a898fb7-f849-4e05-8048-ff7451b1d431 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.537761 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6de684-c45a-4a3d-aa40-5001e2ebfa0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.538646 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6de684-c45a-4a3d-aa40-5001e2ebfa0f admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.538956 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6de684-c45a-4a3d-aa40-5001e2ebfa0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.539277 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6de684-c45a-4a3d-aa40-5001e2ebfa0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.540037 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6de684-c45a-4a3d-aa40-5001e2ebfa0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.548170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6de684-c45a-4a3d-aa40-5001e2ebfa0f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.553652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:20.567371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1228/2444] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:20 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:49:20.569511 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d1a27272f5c43298bbd8c5c29d71dab'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.571585 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6de684-c45a-4a3d-aa40-5001e2ebfa0f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.582527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6de684-c45a-4a3d-aa40-5001e2ebfa0f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.585753 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d1a27272f5c43298bbd8c5c29d71dab'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.586974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d1a27272f5c43298bbd8c5c29d71dab'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.587618 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d178f4-03a6-495c-8ca9-2f8344d9e982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.587995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d1a27272f5c43298bbd8c5c29d71dab'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.588310 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d178f4-03a6-495c-8ca9-2f8344d9e982 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.588596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d178f4-03a6-495c-8ca9-2f8344d9e982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.588871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d178f4-03a6-495c-8ca9-2f8344d9e982 admin admin] PATH_INFO: `/v3/projects/dd5a5d1ce2f840d2ac033637126a5844/users/8bd3c0d70f8e469681655a72225a8c9d/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.601635 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-34b641b0-7d0f-4e95-ae3f-3e44ef697434 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d1a27272f5c43298bbd8c5c29d71dab'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:20.608682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d178f4-03a6-495c-8ca9-2f8344d9e982 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd5a5d1ce2f840d2ac033637126a5844, user_id=8bd3c0d70f8e469681655a72225a8c9d, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.612463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d178f4-03a6-495c-8ca9-2f8344d9e982 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.623521 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:20.634224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1229/2445] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:49:20 2022] PUT /identity/v3/projects/dd5a5d1ce2f840d2ac033637126a5844/users/8bd3c0d70f8e469681655a72225a8c9d/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:49:20.636863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1217/2446] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:49:20 2022] DELETE /identity/v3/users/7d1a27272f5c43298bbd8c5c29d71dab => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:20.643674 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e7d0f7-43b5-43bf-a67f-d51fc0248a13 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.644446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e7d0f7-43b5-43bf-a67f-d51fc0248a13 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.644985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e7d0f7-43b5-43bf-a67f-d51fc0248a13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.646510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba63758e-dcb9-4695-ba15-bbb7e92fa193 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.646813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba63758e-dcb9-4695-ba15-bbb7e92fa193 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.647086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba63758e-dcb9-4695-ba15-bbb7e92fa193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.667243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-49e7d0f7-43b5-43bf-a67f-d51fc0248a13 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:20.668710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-49e7d0f7-43b5-43bf-a67f-d51fc0248a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:20.678795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ba63758e-dcb9-4695-ba15-bbb7e92fa193 None None] MFA Rules not processed for user `8bd3c0d70f8e469681655a72225a8c9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:20.680366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ba63758e-dcb9-4695-ba15-bbb7e92fa193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:20.741754 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1230/2447] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:20.760113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e7d0f7-43b5-43bf-a67f-d51fc0248a13 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.761854 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-49e7d0f7-43b5-43bf-a67f-d51fc0248a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:20.766415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1218/2448] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:20 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:20.782615 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba63758e-dcb9-4695-ba15-bbb7e92fa193 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:20.784446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ba63758e-dcb9-4695-ba15-bbb7e92fa193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:20.855048 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba63758e-dcb9-4695-ba15-bbb7e92fa193 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.858913 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.860125 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.860403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.860665 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.865099 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e7d0f7-43b5-43bf-a67f-d51fc0248a13 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:20.870825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.871831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21ba27f-bc8e-4e75-886e-1f6d9d77c225 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:20.872271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.872905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21ba27f-bc8e-4e75-886e-1f6d9d77c225 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:20.873406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21ba27f-bc8e-4e75-886e-1f6d9d77c225 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:20.873905 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21ba27f-bc8e-4e75-886e-1f6d9d77c225 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:20.882650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1219/2449] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:20 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:49:20.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21ba27f-bc8e-4e75-886e-1f6d9d77c225 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:20.894251 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21ba27f-bc8e-4e75-886e-1f6d9d77c225 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:20.910613 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1231/2450] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:49:21.298024 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:21.299636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:21.342405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7711cb80-0483-4fcd-a9de-2bb1c0223831 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:21.350563 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:21.351884 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:21.352336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:21.352846 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] PATH_INFO: `/v3/projects/65417f6732ff4df4b3e69cc83845f080` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:21.354850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65417f6732ff4df4b3e69cc83845f080)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:21.357050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:21.419813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:21.436064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65417f6732ff4df4b3e69cc83845f080', 'request_id': 'req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:21.449673 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65417f6732ff4df4b3e69cc83845f080', 'request_id': 'req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:21.460713 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65417f6732ff4df4b3e69cc83845f080', 'request_id': 'req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:21.483262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] The token cache is being invalidate because project 65417f6732ff4df4b3e69cc83845f080 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:49:21.483892 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:21.485261 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e9984b30-fdd2-4b01-b12f-92b97b0dd9d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:21.487703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1220/2451] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:49:21 2022] DELETE /identity/v3/projects/65417f6732ff4df4b3e69cc83845f080 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:28.742349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21ba27f-bc8e-4e75-886e-1f6d9d77c225 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:28.753875 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21ba27f-bc8e-4e75-886e-1f6d9d77c225 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:28.812808 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f89ba41-b7b4-477c-a950-30dade2b77a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:28.813427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f89ba41-b7b4-477c-a950-30dade2b77a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:28.813819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f89ba41-b7b4-477c-a950-30dade2b77a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:28.814094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f89ba41-b7b4-477c-a950-30dade2b77a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:28.816023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3f89ba41-b7b4-477c-a950-30dade2b77a8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:28.883997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:28.887473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f89ba41-b7b4-477c-a950-30dade2b77a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:28.889359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f89ba41-b7b4-477c-a950-30dade2b77a8 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:28.901353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1232/2452] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:28 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:49:31.476717 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=11,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:38.331117 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b668625-4a11-4e1c-8051-aa167ef096c5 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:38.332240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b668625-4a11-4e1c-8051-aa167ef096c5 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:38.332240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b668625-4a11-4e1c-8051-aa167ef096c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:38.355293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-5b668625-4a11-4e1c-8051-aa167ef096c5 None None] MFA Rules not processed for user `f738e67d8c2340028da1254b60d7810e`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:38.356734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5b668625-4a11-4e1c-8051-aa167ef096c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:38.371025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1221/2453] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:38 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:38.895915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:39.694967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aae29de-a597-48e0-aed6-f9d68e93b41f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:39.695505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aae29de-a597-48e0-aed6-f9d68e93b41f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:39.695907 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aae29de-a597-48e0-aed6-f9d68e93b41f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:39.719822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9aae29de-a597-48e0-aed6-f9d68e93b41f None None] MFA Rules not processed for user `a48ed2d9ded741ca91afc1fbbd9c608f`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:39.721128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9aae29de-a597-48e0-aed6-f9d68e93b41f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:39.790652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1233/2454] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:41.159382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b668625-4a11-4e1c-8051-aa167ef096c5 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:41.161143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-5b668625-4a11-4e1c-8051-aa167ef096c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:41.283584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b668625-4a11-4e1c-8051-aa167ef096c5 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:41.290936 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:41.292714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:41.292965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:41.293243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] PATH_INFO: `/v3/users/8bd3c0d70f8e469681655a72225a8c9d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:41.294898 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=8bd3c0d70f8e469681655a72225a8c9d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:41.298330 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:41.387030 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8bd3c0d70f8e469681655a72225a8c9d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:41.396391 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8bd3c0d70f8e469681655a72225a8c9d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:41.397179 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8bd3c0d70f8e469681655a72225a8c9d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:41.397910 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8bd3c0d70f8e469681655a72225a8c9d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:41.408741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8bd3c0d70f8e469681655a72225a8c9d'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:41.437876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1222/2455] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:49:41 2022] DELETE /identity/v3/users/8bd3c0d70f8e469681655a72225a8c9d => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:41.444916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0fe709-5b7d-4b06-b938-fd455e972602 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:41.445824 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0fe709-5b7d-4b06-b938-fd455e972602 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:41.445824 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0fe709-5b7d-4b06-b938-fd455e972602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:41.474254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5c0fe709-5b7d-4b06-b938-fd455e972602 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:41.476685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5c0fe709-5b7d-4b06-b938-fd455e972602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:41.628479 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1234/2456] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:41.659934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:41.662246 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:41.733637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9f28e5-920a-4225-b017-b66591911aaa admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:41.738520 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abfaffd-bfdf-4f41-ba62-eb9e9e8e13fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:41.739265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abfaffd-bfdf-4f41-ba62-eb9e9e8e13fb service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:41.739448 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abfaffd-bfdf-4f41-ba62-eb9e9e8e13fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:41.739979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abfaffd-bfdf-4f41-ba62-eb9e9e8e13fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:41.752744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abfaffd-bfdf-4f41-ba62-eb9e9e8e13fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:41.754828 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abfaffd-bfdf-4f41-ba62-eb9e9e8e13fb service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:41.772076 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1223/2457] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:49:42.400255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0fe709-5b7d-4b06-b938-fd455e972602 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:42.402360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5c0fe709-5b7d-4b06-b938-fd455e972602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:42.423681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0fe709-5b7d-4b06-b938-fd455e972602 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:42.432743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:42.433849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:42.434336 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:42.434842 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] PATH_INFO: `/v3/projects/dd5a5d1ce2f840d2ac033637126a5844` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:42.437207 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd5a5d1ce2f840d2ac033637126a5844)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:42.441246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:42.524318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd5a5d1ce2f840d2ac033637126a5844', 'request_id': 'req-a8c60d53-424b-4117-ac07-e0e0129617ac'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:42.535706 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd5a5d1ce2f840d2ac033637126a5844', 'request_id': 'req-a8c60d53-424b-4117-ac07-e0e0129617ac'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:42.559430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd5a5d1ce2f840d2ac033637126a5844', 'request_id': 'req-a8c60d53-424b-4117-ac07-e0e0129617ac'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:42.587832 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] The token cache is being invalidate because project dd5a5d1ce2f840d2ac033637126a5844 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:49:42.588163 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:42.589352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8c60d53-424b-4117-ac07-e0e0129617ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:49:42.591951 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1235/2458] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:49:42 2022] DELETE /identity/v3/projects/dd5a5d1ce2f840d2ac033637126a5844 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:42.598219 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f31df8a-70cd-440c-85e0-8c8e788a56ac None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:42.599306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f31df8a-70cd-440c-85e0-8c8e788a56ac None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:42.599637 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f31df8a-70cd-440c-85e0-8c8e788a56ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:42.642199 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-2f31df8a-70cd-440c-85e0-8c8e788a56ac None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:42.644995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-2f31df8a-70cd-440c-85e0-8c8e788a56ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:42.667063 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:42.757670 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1224/2459] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:42.767012 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6c7e4-bf9e-4e1c-988a-f235b209390a None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:42.767362 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6c7e4-bf9e-4e1c-988a-f235b209390a None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:42.767583 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6c7e4-bf9e-4e1c-988a-f235b209390a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:42.800206 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-65c6c7e4-bf9e-4e1c-988a-f235b209390a None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:42.801775 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-65c6c7e4-bf9e-4e1c-988a-f235b209390a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:42.822044 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1236/2460] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:49:42.824200 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f31df8a-70cd-440c-85e0-8c8e788a56ac None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:42.840271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f31df8a-70cd-440c-85e0-8c8e788a56ac None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:42.848119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:42.848705 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:42.849027 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:42.849281 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:42.849934 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:49:42.850295 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:42.853155 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:42.870801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1225/2461] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:49:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:49:42.877584 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c6c7e4-bf9e-4e1c-988a-f235b209390a None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:42.895685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c6c7e4-bf9e-4e1c-988a-f235b209390a None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:42.902005 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503b02af-16a7-4487-ae8f-fc3c8c4297cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:42.903190 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503b02af-16a7-4487-ae8f-fc3c8c4297cc admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:42.903644 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503b02af-16a7-4487-ae8f-fc3c8c4297cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:42.904237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503b02af-16a7-4487-ae8f-fc3c8c4297cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:42.905897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503b02af-16a7-4487-ae8f-fc3c8c4297cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:42.912224 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503b02af-16a7-4487-ae8f-fc3c8c4297cc admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:42.970331 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:42.990199 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1237/2462] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:49:42 2022] POST /identity/v3/projects => generated 383 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:49:42.998887 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:43.023134 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1744191-b2db-46bf-b44a-f3eb93db725d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:43.159080 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99e8de-3d47-4f59-885f-03aafd010bc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:43.160265 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99e8de-3d47-4f59-885f-03aafd010bc2 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:43.160774 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99e8de-3d47-4f59-885f-03aafd010bc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:43.161440 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99e8de-3d47-4f59-885f-03aafd010bc2 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:43.163004 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d99e8de-3d47-4f59-885f-03aafd010bc2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:43.166961 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d99e8de-3d47-4f59-885f-03aafd010bc2 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:43.228803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1226/2463] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:49:42 2022] POST /identity/v3/users => generated 292 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:49:43.229958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503b02af-16a7-4487-ae8f-fc3c8c4297cc admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:43.248303 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503b02af-16a7-4487-ae8f-fc3c8c4297cc admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:43.254650 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061b8e7b-f755-4bfe-a073-da4d846c888e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:43.255200 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b8e7b-f755-4bfe-a073-da4d846c888e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:43.255482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b8e7b-f755-4bfe-a073-da4d846c888e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:43.255767 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b8e7b-f755-4bfe-a073-da4d846c888e admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:43.256574 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061b8e7b-f755-4bfe-a073-da4d846c888e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:43.258316 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061b8e7b-f755-4bfe-a073-da4d846c888e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:43.276146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1238/2464] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:43 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:49:43.279177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99e8de-3d47-4f59-885f-03aafd010bc2 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:43.293463 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99e8de-3d47-4f59-885f-03aafd010bc2 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:43.304412 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867f69ad-9b42-4fd8-beb7-aa714ecfd6c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:43.305874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f69ad-9b42-4fd8-beb7-aa714ecfd6c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:43.306585 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f69ad-9b42-4fd8-beb7-aa714ecfd6c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:43.307165 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f69ad-9b42-4fd8-beb7-aa714ecfd6c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:43.308517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867f69ad-9b42-4fd8-beb7-aa714ecfd6c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:43.313167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867f69ad-9b42-4fd8-beb7-aa714ecfd6c1 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:43.348037 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1227/2465] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:43 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:49:43.354015 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061b8e7b-f755-4bfe-a073-da4d846c888e admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:43.374302 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061b8e7b-f755-4bfe-a073-da4d846c888e admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:43.386018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:43.387435 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:43.388019 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:43.388568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] PATH_INFO: `/v3/projects/a8a6d900d9d344a694b955f51fa46aa9/users/531efd120096452caf8c0b7a7f6b9ba1/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:43.423656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8a6d900d9d344a694b955f51fa46aa9, user_id=531efd120096452caf8c0b7a7f6b9ba1, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:43.430789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:43.459359 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1239/2466] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:49:43 2022] PUT /identity/v3/projects/a8a6d900d9d344a694b955f51fa46aa9/users/531efd120096452caf8c0b7a7f6b9ba1/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:43.467431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478314fb-274a-4f57-ab07-68737466edd2 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:43.468471 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478314fb-274a-4f57-ab07-68737466edd2 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:43.469236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478314fb-274a-4f57-ab07-68737466edd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:43.516073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-478314fb-274a-4f57-ab07-68737466edd2 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:43.516073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-478314fb-274a-4f57-ab07-68737466edd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:43.644371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1228/2467] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:43.656900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:43.659197 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:43.750272 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e176168-503f-4e3d-b8ba-9b6ded6ca080 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:43.757490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0279e99-c544-428c-87db-9f4683884053 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:43.758602 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0279e99-c544-428c-87db-9f4683884053 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:43.759091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0279e99-c544-428c-87db-9f4683884053 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:43.759563 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0279e99-c544-428c-87db-9f4683884053 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:43.779960 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0279e99-c544-428c-87db-9f4683884053 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:43.783249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0279e99-c544-428c-87db-9f4683884053 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:43.802018 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1240/2468] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:49:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:49:51.208663 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd2560-c22b-4b19-b029-7ded471386ef None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:51.209400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd2560-c22b-4b19-b029-7ded471386ef None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:51.209725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd2560-c22b-4b19-b029-7ded471386ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:51.273404 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-96fd2560-c22b-4b19-b029-7ded471386ef None None] MFA Rules not processed for user `531efd120096452caf8c0b7a7f6b9ba1`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:51.276551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-96fd2560-c22b-4b19-b029-7ded471386ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:51.297038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:49:51.434346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1229/2469] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:49:51.440776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0279e99-c544-428c-87db-9f4683884053 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:51.461596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0279e99-c544-428c-87db-9f4683884053 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:51.469355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9a42d0-cd45-4ec2-b86d-2c51f7b08d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:51.470344 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9a42d0-cd45-4ec2-b86d-2c51f7b08d14 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:51.471355 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9a42d0-cd45-4ec2-b86d-2c51f7b08d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:51.472581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9a42d0-cd45-4ec2-b86d-2c51f7b08d14 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:51.473796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9a42d0-cd45-4ec2-b86d-2c51f7b08d14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:51.476039 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9a42d0-cd45-4ec2-b86d-2c51f7b08d14 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:51.548933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1241/2470] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:49:51 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:49:51.549544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fd2560-c22b-4b19-b029-7ded471386ef None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:51.583139 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fd2560-c22b-4b19-b029-7ded471386ef None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:51.767771 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a205e1c-fd08-4831-ad78-ccc2b519ec01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:51.768815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a205e1c-fd08-4831-ad78-ccc2b519ec01 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:51.769314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a205e1c-fd08-4831-ad78-ccc2b519ec01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:51.769915 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a205e1c-fd08-4831-ad78-ccc2b519ec01 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:51.773594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a205e1c-fd08-4831-ad78-ccc2b519ec01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:51.773594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a205e1c-fd08-4831-ad78-ccc2b519ec01 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:51.818078 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1230/2471] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:49:51 2022] POST /identity/v3/users => generated 299 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:49:51.822725 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9a42d0-cd45-4ec2-b86d-2c51f7b08d14 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:51.838872 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9a42d0-cd45-4ec2-b86d-2c51f7b08d14 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:51.847385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2eed736-a957-4a15-a8f8-cbc03d5259a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:51.848036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eed736-a957-4a15-a8f8-cbc03d5259a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:51.848318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eed736-a957-4a15-a8f8-cbc03d5259a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:51.848612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eed736-a957-4a15-a8f8-cbc03d5259a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:51.849298 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2eed736-a957-4a15-a8f8-cbc03d5259a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:51.851320 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2eed736-a957-4a15-a8f8-cbc03d5259a1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:51.867247 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1242/2472] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:51 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:49:51.872653 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a205e1c-fd08-4831-ad78-ccc2b519ec01 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:51.896984 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a205e1c-fd08-4831-ad78-ccc2b519ec01 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:51.907507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a40aab-c399-4e7f-b425-d14a6fa37256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:51.908908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a40aab-c399-4e7f-b425-d14a6fa37256 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:51.909562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a40aab-c399-4e7f-b425-d14a6fa37256 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:51.910182 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a40aab-c399-4e7f-b425-d14a6fa37256 admin admin] PATH_INFO: `/v3/projects/c3dc487565414290a5b5fa5c68e1c100/users/25afec59b30d4079a1e9bc73c6433902/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:51.944571 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a40aab-c399-4e7f-b425-d14a6fa37256 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3dc487565414290a5b5fa5c68e1c100, user_id=25afec59b30d4079a1e9bc73c6433902, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:51.949965 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a40aab-c399-4e7f-b425-d14a6fa37256 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:51.984634 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2eed736-a957-4a15-a8f8-cbc03d5259a1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:51.985411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1231/2473] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:49:51 2022] PUT /identity/v3/projects/c3dc487565414290a5b5fa5c68e1c100/users/25afec59b30d4079a1e9bc73c6433902/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:49:52.000480 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2eed736-a957-4a15-a8f8-cbc03d5259a1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:52.098291 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247ca421-7566-400e-9aec-57a5e055e432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:52.098962 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247ca421-7566-400e-9aec-57a5e055e432 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:52.099257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247ca421-7566-400e-9aec-57a5e055e432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:52.099540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247ca421-7566-400e-9aec-57a5e055e432 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:52.100220 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247ca421-7566-400e-9aec-57a5e055e432 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:52.101900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247ca421-7566-400e-9aec-57a5e055e432 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:52.114888 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1243/2474] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:49:51 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:49:52.119740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a40aab-c399-4e7f-b425-d14a6fa37256 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:49:52.134143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a40aab-c399-4e7f-b425-d14a6fa37256 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:49:52.143147 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:49:52.144363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:52.144850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:52.146260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] PATH_INFO: `/v3/projects/c3dc487565414290a5b5fa5c68e1c100/users/25afec59b30d4079a1e9bc73c6433902/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:52.149952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3dc487565414290a5b5fa5c68e1c100, user_id=25afec59b30d4079a1e9bc73c6433902, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:49:52.155562 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:49:52.172716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1232/2475] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:49:52 2022] PUT /identity/v3/projects/c3dc487565414290a5b5fa5c68e1c100/users/25afec59b30d4079a1e9bc73c6433902/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:49:59.755818 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e39232-4d64-41c4-996f-23367ea5f5ae None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:49:59.756705 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e39232-4d64-41c4-996f-23367ea5f5ae None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:49:59.757174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e39232-4d64-41c4-996f-23367ea5f5ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:49:59.846900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-15e39232-4d64-41c4-996f-23367ea5f5ae None None] MFA Rules not processed for user `25afec59b30d4079a1e9bc73c6433902`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:49:59.851615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-15e39232-4d64-41c4-996f-23367ea5f5ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:49:59.931127 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:50:00.130187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1244/2476] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:49:59 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 371 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:50:00.168099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:50:00.172517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:50:00.400833 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3313ec4b-1243-490c-8f54-f9db07383870 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:50:00.414953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e3f6be584e34bfb9e430c7756d071d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:50:00.416296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:00.416688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:00.417061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:00.467278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:50:00.562038 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:50:00.563860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:50:00.581810 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1233/2477] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:50:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:50:00.866606 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16408ea6-d315-43ff-bb7d-4023bac02757 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:00.867189 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16408ea6-d315-43ff-bb7d-4023bac02757 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:00.867569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16408ea6-d315-43ff-bb7d-4023bac02757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:00.896052 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-16408ea6-d315-43ff-bb7d-4023bac02757 None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:50:00.897525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-16408ea6-d315-43ff-bb7d-4023bac02757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:50:01.000587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1245/2478] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:50:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:50:01.016852 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:50:01.018367 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:50:01.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed247163-3a1e-48a9-bd4b-3cc8ec27c709 service glance] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:50:01.145627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad227510-bc0e-4bcd-961e-c251385b0d97 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:50:01.146456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad227510-bc0e-4bcd-961e-c251385b0d97 service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:01.146870 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad227510-bc0e-4bcd-961e-c251385b0d97 service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:01.147263 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad227510-bc0e-4bcd-961e-c251385b0d97 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:01.160297 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad227510-bc0e-4bcd-961e-c251385b0d97 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:50:01.163798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad227510-bc0e-4bcd-961e-c251385b0d97 service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:50:01.183495 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1234/2479] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:50:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:50:03.084334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16408ea6-d315-43ff-bb7d-4023bac02757 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:50:03.086849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-16408ea6-d315-43ff-bb7d-4023bac02757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:50:03.229512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16408ea6-d315-43ff-bb7d-4023bac02757 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:50:03.238525 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c25c31-a4ff-4d0e-9581-d19e5e9fd52c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:50:03.240063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c25c31-a4ff-4d0e-9581-d19e5e9fd52c service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:03.241091 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c25c31-a4ff-4d0e-9581-d19e5e9fd52c service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:03.241843 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c25c31-a4ff-4d0e-9581-d19e5e9fd52c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:03.245382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c5c25c31-a4ff-4d0e-9581-d19e5e9fd52c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:50:03.383937 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c25c31-a4ff-4d0e-9581-d19e5e9fd52c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:50:03.386726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c25c31-a4ff-4d0e-9581-d19e5e9fd52c service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:50:03.409490 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1246/2480] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:50:03 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:50:11.174072 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=51 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:50:13.401697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:50:14.059880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a25cae-d59c-43d7-95e7-6a125c4fe92e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:14.061026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a25cae-d59c-43d7-95e7-6a125c4fe92e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:14.061026 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a25cae-d59c-43d7-95e7-6a125c4fe92e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:14.082407 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-64a25cae-d59c-43d7-95e7-6a125c4fe92e None None] MFA Rules not processed for user `531efd120096452caf8c0b7a7f6b9ba1`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:50:14.083691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-64a25cae-d59c-43d7-95e7-6a125c4fe92e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:50:14.090985 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1235/2481] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:50:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:50:16.720314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641aed8-aee0-49de-bd5f-26f864918727 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:16.721034 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641aed8-aee0-49de-bd5f-26f864918727 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:16.721827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e641aed8-aee0-49de-bd5f-26f864918727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:16.750664 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e641aed8-aee0-49de-bd5f-26f864918727 None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:50:16.751946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e641aed8-aee0-49de-bd5f-26f864918727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:50:16.760587 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1247/2482] 10.222.0.26 () {58 vars in 1009 bytes} [Wed Oct 19 00:50:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:50:16.788362 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a25cae-d59c-43d7-95e7-6a125c4fe92e None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:50:16.802960 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a25cae-d59c-43d7-95e7-6a125c4fe92e None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:50:16.809522 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912438e9-8ee7-41c3-9019-d09ab2117981 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:50:16.810239 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912438e9-8ee7-41c3-9019-d09ab2117981 service swift] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:16.810699 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912438e9-8ee7-41c3-9019-d09ab2117981 service swift] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:16.811071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912438e9-8ee7-41c3-9019-d09ab2117981 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:16.821057 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912438e9-8ee7-41c3-9019-d09ab2117981 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:50:16.822744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912438e9-8ee7-41c3-9019-d09ab2117981 service swift] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:50:16.833390 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1236/2483] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:50:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:50:24.871503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e641aed8-aee0-49de-bd5f-26f864918727 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:50:24.889811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e641aed8-aee0-49de-bd5f-26f864918727 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:50:24.896558 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:50:24.897586 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:24.898060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:24.898571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:24.901229 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:50:25.040608 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:50:25.042158 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:50:25.068129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1248/2484] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:50:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:50:26.829895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=9 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:50:35.064709 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=46 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:50:45.911525 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912438e9-8ee7-41c3-9019-d09ab2117981 service swift] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:50:45.923657 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912438e9-8ee7-41c3-9019-d09ab2117981 service swift] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:50:45.928369 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247cf83f-2402-4ce8-8945-2e33cb49424d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:50:45.929067 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cf83f-2402-4ce8-8945-2e33cb49424d service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:50:45.929406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cf83f-2402-4ce8-8945-2e33cb49424d service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:50:45.929745 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cf83f-2402-4ce8-8945-2e33cb49424d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:50:45.942309 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247cf83f-2402-4ce8-8945-2e33cb49424d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:50:45.944104 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247cf83f-2402-4ce8-8945-2e33cb49424d service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:50:45.958624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1237/2485] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:50:45 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:50:55.950759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:10.135371 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:10.138915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:10.169410 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dadfa7-aa5e-4b3e-acb3-14d27f9afb34 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:10.180285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:10.182958 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:10.183930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:10.184406 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:10.187819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:10.219792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:10.223237 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:10.250195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1249/2486] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:16.306661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247cf83f-2402-4ce8-8945-2e33cb49424d service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:16.322306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247cf83f-2402-4ce8-8945-2e33cb49424d service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:16.329328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:16.330254 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:16.330719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:16.331170 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:16.334159 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:16.440831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:16.442216 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:16.454882 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1238/2487] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:20.238889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=6 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:21.273708 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:21.275496 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:21.369569 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:21.372580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:21.403231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20941727-cc37-47a3-8970-1714fcb520de service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:21.411829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:21.412868 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:21.413268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:21.413597 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] PATH_INFO: `/v3/users/a48ed2d9ded741ca91afc1fbbd9c608f` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:21.415816 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] RBAC: Authorizing `identity:delete_user(user_id=a48ed2d9ded741ca91afc1fbbd9c608f)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:21.419094 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:21.425075 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0789d7-31f8-41f2-a786-f6395c7a66db service nova] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:21.432314 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:21.433790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:21.434944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:21.434944 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:21.436968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:21.452485 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:21.454542 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:21.469419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1250/2488] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:21.516576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a48ed2d9ded741ca91afc1fbbd9c608f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:21.525661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a48ed2d9ded741ca91afc1fbbd9c608f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:21.526701 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a48ed2d9ded741ca91afc1fbbd9c608f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:21.527556 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a48ed2d9ded741ca91afc1fbbd9c608f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:21.537032 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6df70ee4-07ef-496b-85f7-2e94fdcde50f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a48ed2d9ded741ca91afc1fbbd9c608f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:21.566035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1239/2489] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:51:21 2022] DELETE /identity/v3/users/a48ed2d9ded741ca91afc1fbbd9c608f => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:51:21.819511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:21.840063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49cc5c1-1854-4d5a-94da-17c522d0eb49 service placement] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:21.928688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:21.929351 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:21.929633 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:21.929912 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] PATH_INFO: `/v3/projects/82d7723ef95b4c3faeac9835eeb233a5` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:21.931254 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=82d7723ef95b4c3faeac9835eeb233a5)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:21.933952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:21.992419 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82d7723ef95b4c3faeac9835eeb233a5', 'request_id': 'req-1a52ce58-0abc-496a-a472-6fae55d8d8ea'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:21.999534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82d7723ef95b4c3faeac9835eeb233a5', 'request_id': 'req-1a52ce58-0abc-496a-a472-6fae55d8d8ea'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:22.008399 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82d7723ef95b4c3faeac9835eeb233a5', 'request_id': 'req-1a52ce58-0abc-496a-a472-6fae55d8d8ea'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:22.022392 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] The token cache is being invalidate because project 82d7723ef95b4c3faeac9835eeb233a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:51:22.022768 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:22.023393 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-1a52ce58-0abc-496a-a472-6fae55d8d8ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:22.025058 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1251/2490] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:51:21 2022] DELETE /identity/v3/projects/82d7723ef95b4c3faeac9835eeb233a5 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:22.032193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ae7cc-7809-47a6-b9c0-cdb8686c29e3 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.032575 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ae7cc-7809-47a6-b9c0-cdb8686c29e3 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.032944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ae7cc-7809-47a6-b9c0-cdb8686c29e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.056002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-778ae7cc-7809-47a6-b9c0-cdb8686c29e3 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:22.057352 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-778ae7cc-7809-47a6-b9c0-cdb8686c29e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:22.157692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1240/2491] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:22.169081 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886a5e00-b850-4794-ae1b-628513fdc0f9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.169529 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886a5e00-b850-4794-ae1b-628513fdc0f9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.169946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886a5e00-b850-4794-ae1b-628513fdc0f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.203810 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-886a5e00-b850-4794-ae1b-628513fdc0f9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:22.206024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-886a5e00-b850-4794-ae1b-628513fdc0f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:22.221825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1252/2492] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:22.226899 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778ae7cc-7809-47a6-b9c0-cdb8686c29e3 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:22.239760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778ae7cc-7809-47a6-b9c0-cdb8686c29e3 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:22.245101 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:22.246193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.246799 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.247174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.247957 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:51:22.248357 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:22.250236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:22.252636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:22.272848 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1241/2493] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:51:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:51:22.280004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886a5e00-b850-4794-ae1b-628513fdc0f9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:22.294269 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886a5e00-b850-4794-ae1b-628513fdc0f9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:22.299570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c3487-a164-474c-846d-6f0db069f7c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:22.300210 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c3487-a164-474c-846d-6f0db069f7c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.300493 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c3487-a164-474c-846d-6f0db069f7c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.300770 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c3487-a164-474c-846d-6f0db069f7c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.301436 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c3487-a164-474c-846d-6f0db069f7c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:22.303451 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c3487-a164-474c-846d-6f0db069f7c7 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:22.318287 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:22.353412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1253/2494] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:51:22 2022] POST /identity/v3/projects => generated 381 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:22.358431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:22.378028 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489fc651-9862-498a-ba47-7a6f33a3df0a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:22.460909 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a478066-67db-4e1b-926c-a638e5dbc251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:22.461879 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a478066-67db-4e1b-926c-a638e5dbc251 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.462325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a478066-67db-4e1b-926c-a638e5dbc251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.462815 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a478066-67db-4e1b-926c-a638e5dbc251 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.463876 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a478066-67db-4e1b-926c-a638e5dbc251 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:22.468017 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a478066-67db-4e1b-926c-a638e5dbc251 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:22.513257 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1242/2495] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:51:22 2022] POST /identity/v3/users => generated 291 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:22.515064 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c3487-a164-474c-846d-6f0db069f7c7 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:22.527599 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c3487-a164-474c-846d-6f0db069f7c7 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:22.533319 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1afad-192d-481f-a7f8-2e605ed41b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:22.534105 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1afad-192d-481f-a7f8-2e605ed41b1b admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.534504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1afad-192d-481f-a7f8-2e605ed41b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.534996 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1afad-192d-481f-a7f8-2e605ed41b1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.535946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e1afad-192d-481f-a7f8-2e605ed41b1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:22.538103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e1afad-192d-481f-a7f8-2e605ed41b1b admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:22.557188 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1254/2496] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:22 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:51:22.561987 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a478066-67db-4e1b-926c-a638e5dbc251 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:22.575127 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a478066-67db-4e1b-926c-a638e5dbc251 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:22.580761 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a89bb0-f86c-4ec7-a59e-d3342cccd4ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:22.581397 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a89bb0-f86c-4ec7-a59e-d3342cccd4ea admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.581661 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a89bb0-f86c-4ec7-a59e-d3342cccd4ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.582035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a89bb0-f86c-4ec7-a59e-d3342cccd4ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.582747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a89bb0-f86c-4ec7-a59e-d3342cccd4ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:22.584539 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a89bb0-f86c-4ec7-a59e-d3342cccd4ea admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:22.598053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1243/2497] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:22 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:51:22.602874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1afad-192d-481f-a7f8-2e605ed41b1b admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:22.615118 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1afad-192d-481f-a7f8-2e605ed41b1b admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:22.621309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:22.622353 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.622796 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.623308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] PATH_INFO: `/v3/projects/7e8918ddd84e4f1a9c3e0003ae181c1a/users/dc1a31109df44058a2f739f310ddf87b/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.642816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e8918ddd84e4f1a9c3e0003ae181c1a, user_id=dc1a31109df44058a2f739f310ddf87b, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:22.645579 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:22.662448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1255/2498] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:22 2022] PUT /identity/v3/projects/7e8918ddd84e4f1a9c3e0003ae181c1a/users/dc1a31109df44058a2f739f310ddf87b/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:51:22.673349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac059241-77e8-4c59-a7be-19bf2699c327 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.673634 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac059241-77e8-4c59-a7be-19bf2699c327 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.673918 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac059241-77e8-4c59-a7be-19bf2699c327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.708908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-ac059241-77e8-4c59-a7be-19bf2699c327 None None] MFA Rules not processed for user `dc1a31109df44058a2f739f310ddf87b`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:22.711233 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-ac059241-77e8-4c59-a7be-19bf2699c327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:22.796429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1244/2499] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:22 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:51:22.807285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:22.809744 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:22.893033 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99984279-94f4-4245-b013-b8b00a60ec77 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:22.898976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c783ba-995a-4ba9-bffd-04db3ec0367a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:22.899616 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c783ba-995a-4ba9-bffd-04db3ec0367a service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:22.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c783ba-995a-4ba9-bffd-04db3ec0367a service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:22.900170 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c783ba-995a-4ba9-bffd-04db3ec0367a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:22.909510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c783ba-995a-4ba9-bffd-04db3ec0367a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:22.911179 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c783ba-995a-4ba9-bffd-04db3ec0367a service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:22.921924 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1256/2500] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:22 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:30.535393 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac059241-77e8-4c59-a7be-19bf2699c327 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:30.634329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac059241-77e8-4c59-a7be-19bf2699c327 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:30.639656 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:30.640307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:30.640580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:30.640850 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] PATH_INFO: `/v3/users/dc1a31109df44058a2f739f310ddf87b` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:30.642053 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc1a31109df44058a2f739f310ddf87b)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:30.644065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:30.647033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:30.723977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc1a31109df44058a2f739f310ddf87b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:30.738675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc1a31109df44058a2f739f310ddf87b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:30.739791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc1a31109df44058a2f739f310ddf87b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:30.740726 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc1a31109df44058a2f739f310ddf87b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:30.753528 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc1a31109df44058a2f739f310ddf87b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:30.775594 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1245/2501] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:51:30 2022] DELETE /identity/v3/users/dc1a31109df44058a2f739f310ddf87b => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:51:30.781028 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd66cae-9324-44b7-87da-50257354f62c None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:30.781580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd66cae-9324-44b7-87da-50257354f62c None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:30.781580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd66cae-9324-44b7-87da-50257354f62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:30.804494 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-2dd66cae-9324-44b7-87da-50257354f62c None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:30.805916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2dd66cae-9324-44b7-87da-50257354f62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:30.880985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1257/2502] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:30.892578 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:30.894279 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:30.966709 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa94551-6523-4335-8d37-486997f24f6c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:30.971314 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa6cff3-2fec-4b51-93fa-250af9e4594f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:30.972262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa6cff3-2fec-4b51-93fa-250af9e4594f service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:30.972547 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa6cff3-2fec-4b51-93fa-250af9e4594f service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:30.972873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa6cff3-2fec-4b51-93fa-250af9e4594f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:30.983622 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa6cff3-2fec-4b51-93fa-250af9e4594f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:30.985398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa6cff3-2fec-4b51-93fa-250af9e4594f service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:30.998268 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1246/2503] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:31.648546 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd66cae-9324-44b7-87da-50257354f62c None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:31.652527 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2dd66cae-9324-44b7-87da-50257354f62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:31.674825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd66cae-9324-44b7-87da-50257354f62c None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:31.684868 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:31.685889 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:31.686325 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:31.686806 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] PATH_INFO: `/v3/projects/7e8918ddd84e4f1a9c3e0003ae181c1a` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:31.688615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e8918ddd84e4f1a9c3e0003ae181c1a)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:31.691068 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:31.738409 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:31.760523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e8918ddd84e4f1a9c3e0003ae181c1a', 'request_id': 'req-7fe08966-0772-49ae-9254-5f2ad7744091'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:31.768036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e8918ddd84e4f1a9c3e0003ae181c1a', 'request_id': 'req-7fe08966-0772-49ae-9254-5f2ad7744091'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:31.778919 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e8918ddd84e4f1a9c3e0003ae181c1a', 'request_id': 'req-7fe08966-0772-49ae-9254-5f2ad7744091'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:31.792900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] The token cache is being invalidate because project 7e8918ddd84e4f1a9c3e0003ae181c1a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:51:31.794057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:31.794057 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7fe08966-0772-49ae-9254-5f2ad7744091 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:31.795748 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1258/2504] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:51:31 2022] DELETE /identity/v3/projects/7e8918ddd84e4f1a9c3e0003ae181c1a => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:31.802084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732de55-104c-4acc-8214-5df7754c4adf None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:31.802733 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732de55-104c-4acc-8214-5df7754c4adf None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:31.803084 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1732de55-104c-4acc-8214-5df7754c4adf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:31.840372 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-1732de55-104c-4acc-8214-5df7754c4adf None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:31.842602 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1732de55-104c-4acc-8214-5df7754c4adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:31.930654 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1247/2505] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:31.942685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3300d4f-319d-4bc9-820c-6da42edf5c9f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:31.943187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3300d4f-319d-4bc9-820c-6da42edf5c9f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:31.943682 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3300d4f-319d-4bc9-820c-6da42edf5c9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:31.970620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-e3300d4f-319d-4bc9-820c-6da42edf5c9f None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:31.972107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e3300d4f-319d-4bc9-820c-6da42edf5c9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:31.980930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1259/2506] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:31.985945 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1732de55-104c-4acc-8214-5df7754c4adf None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:31.997517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1732de55-104c-4acc-8214-5df7754c4adf None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.002445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.003124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.003398 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.003664 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] PATH_INFO: `/v3/domains` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.004351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81812) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:51:32.004650 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.006271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.020773 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1248/2507] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:51:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 00:51:32.025478 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3300d4f-319d-4bc9-820c-6da42edf5c9f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.035492 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3300d4f-319d-4bc9-820c-6da42edf5c9f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.040844 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53580d10-08a5-40b3-b441-5be3466f5690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.041568 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53580d10-08a5-40b3-b441-5be3466f5690 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.041906 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53580d10-08a5-40b3-b441-5be3466f5690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.042266 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53580d10-08a5-40b3-b441-5be3466f5690 admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.043006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53580d10-08a5-40b3-b441-5be3466f5690 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.045156 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53580d10-08a5-40b3-b441-5be3466f5690 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.079122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1260/2508] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:51:32 2022] POST /identity/v3/projects => generated 409 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:32.083891 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.095527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8aac00-8b54-47dc-986d-afa7ebada71f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.105494 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:32.168381 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fc9e74-2d4b-462f-acb9-f925e331b17a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.169025 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc9e74-2d4b-462f-acb9-f925e331b17a admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.169403 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc9e74-2d4b-462f-acb9-f925e331b17a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.169716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc9e74-2d4b-462f-acb9-f925e331b17a admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.170405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fc9e74-2d4b-462f-acb9-f925e331b17a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.172861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fc9e74-2d4b-462f-acb9-f925e331b17a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.199157 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1249/2509] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:51:32 2022] POST /identity/v3/users => generated 305 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:51:32.204201 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53580d10-08a5-40b3-b441-5be3466f5690 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.215820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53580d10-08a5-40b3-b441-5be3466f5690 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.221360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b6d7cf-e0a3-4179-870f-14540a0e87d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.222030 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b6d7cf-e0a3-4179-870f-14540a0e87d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.222511 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b6d7cf-e0a3-4179-870f-14540a0e87d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.222966 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b6d7cf-e0a3-4179-870f-14540a0e87d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.223722 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b6d7cf-e0a3-4179-870f-14540a0e87d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.225686 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b6d7cf-e0a3-4179-870f-14540a0e87d2 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.239395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1261/2510] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:51:32.244079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fc9e74-2d4b-462f-acb9-f925e331b17a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.256382 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fc9e74-2d4b-462f-acb9-f925e331b17a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.264162 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c682f44-2e4c-4783-b00e-8e23341ca0af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.264952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c682f44-2e4c-4783-b00e-8e23341ca0af admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.265322 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c682f44-2e4c-4783-b00e-8e23341ca0af admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.265708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c682f44-2e4c-4783-b00e-8e23341ca0af admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.266584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c682f44-2e4c-4783-b00e-8e23341ca0af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.268491 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c682f44-2e4c-4783-b00e-8e23341ca0af admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.282259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1250/2511] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:32 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:51:32.288943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b6d7cf-e0a3-4179-870f-14540a0e87d2 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.305491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b6d7cf-e0a3-4179-870f-14540a0e87d2 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.312154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6536e86-550a-4c5f-986c-3d716f308509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.312915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6536e86-550a-4c5f-986c-3d716f308509 admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.313323 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6536e86-550a-4c5f-986c-3d716f308509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.313649 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6536e86-550a-4c5f-986c-3d716f308509 admin admin] PATH_INFO: `/v3/projects/70232ea3c5c443f9ac810901872889ed/users/4602605c0a6949a6b6833d46c9b7df15/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.333129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6536e86-550a-4c5f-986c-3d716f308509 admin admin] RBAC: Authorizing `identity:create_grant(project_id=70232ea3c5c443f9ac810901872889ed, user_id=4602605c0a6949a6b6833d46c9b7df15, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.336487 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6536e86-550a-4c5f-986c-3d716f308509 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.357873 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1262/2512] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:32 2022] PUT /identity/v3/projects/70232ea3c5c443f9ac810901872889ed/users/4602605c0a6949a6b6833d46c9b7df15/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:32.375294 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759998e5-3f10-4381-bb10-ded96a7542af None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.375636 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759998e5-3f10-4381-bb10-ded96a7542af None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.375928 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759998e5-3f10-4381-bb10-ded96a7542af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.409388 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-759998e5-3f10-4381-bb10-ded96a7542af None None] MFA Rules not processed for user `4602605c0a6949a6b6833d46c9b7df15`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:32.410703 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-759998e5-3f10-4381-bb10-ded96a7542af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:32.528830 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1251/2513] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:32 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:32.534657 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6536e86-550a-4c5f-986c-3d716f308509 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.651824 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6536e86-550a-4c5f-986c-3d716f308509 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.659622 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49847fa6-2480-4bed-9963-0f8a67aa1acc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.660676 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49847fa6-2480-4bed-9963-0f8a67aa1acc admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.661140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49847fa6-2480-4bed-9963-0f8a67aa1acc admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.661611 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49847fa6-2480-4bed-9963-0f8a67aa1acc admin admin] PATH_INFO: `/v3/projects` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.662802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49847fa6-2480-4bed-9963-0f8a67aa1acc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.666570 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49847fa6-2480-4bed-9963-0f8a67aa1acc admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.710576 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1263/2514] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:51:32 2022] POST /identity/v3/projects => generated 407 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:32.715400 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759998e5-3f10-4381-bb10-ded96a7542af None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.727866 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759998e5-3f10-4381-bb10-ded96a7542af None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.826728 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3a334f-43f0-4bc6-97ac-3c60229b5570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.827552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3a334f-43f0-4bc6-97ac-3c60229b5570 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.827763 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3a334f-43f0-4bc6-97ac-3c60229b5570 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.828142 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3a334f-43f0-4bc6-97ac-3c60229b5570 admin admin] PATH_INFO: `/v3/users` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.828999 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3a334f-43f0-4bc6-97ac-3c60229b5570 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.831262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3a334f-43f0-4bc6-97ac-3c60229b5570 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.857236 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1252/2515] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:51:32 2022] POST /identity/v3/users => generated 310 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:32.862195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49847fa6-2480-4bed-9963-0f8a67aa1acc admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.875025 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49847fa6-2480-4bed-9963-0f8a67aa1acc admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.880172 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f897550-c061-4cda-8b65-ffd0497a41ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.880861 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f897550-c061-4cda-8b65-ffd0497a41ad admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.881226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f897550-c061-4cda-8b65-ffd0497a41ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.881547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f897550-c061-4cda-8b65-ffd0497a41ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.882322 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f897550-c061-4cda-8b65-ffd0497a41ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.884457 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f897550-c061-4cda-8b65-ffd0497a41ad admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.897477 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1264/2516] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:51:32.903671 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3a334f-43f0-4bc6-97ac-3c60229b5570 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.914456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3a334f-43f0-4bc6-97ac-3c60229b5570 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:32.920526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a392aa29-7d65-4306-a009-a10e05bd0ad5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:32.921203 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a392aa29-7d65-4306-a009-a10e05bd0ad5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:32.921479 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a392aa29-7d65-4306-a009-a10e05bd0ad5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:32.921751 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a392aa29-7d65-4306-a009-a10e05bd0ad5 admin admin] PATH_INFO: `/v3/projects/a1d98123a2bf4cf09e26d057ff7bc316/users/e3c4b851c4504e76aaca747d1fa0fcfb/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:32.925948 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:32.943192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a392aa29-7d65-4306-a009-a10e05bd0ad5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1d98123a2bf4cf09e26d057ff7bc316, user_id=e3c4b851c4504e76aaca747d1fa0fcfb, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:32.946029 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a392aa29-7d65-4306-a009-a10e05bd0ad5 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:32.963308 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1253/2517] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:32 2022] PUT /identity/v3/projects/a1d98123a2bf4cf09e26d057ff7bc316/users/e3c4b851c4504e76aaca747d1fa0fcfb/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:51:32.967858 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f897550-c061-4cda-8b65-ffd0497a41ad admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:32.980491 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f897550-c061-4cda-8b65-ffd0497a41ad admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:33.012445 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:33.056998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46bbde7-f568-4c41-8ce7-d320940d8707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:33.058271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46bbde7-f568-4c41-8ce7-d320940d8707 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:33.058596 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46bbde7-f568-4c41-8ce7-d320940d8707 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:33.058871 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46bbde7-f568-4c41-8ce7-d320940d8707 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:33.059631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46bbde7-f568-4c41-8ce7-d320940d8707 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:33.061366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46bbde7-f568-4c41-8ce7-d320940d8707 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:33.074964 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1265/2518] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:32 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:51:33.080158 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a392aa29-7d65-4306-a009-a10e05bd0ad5 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:33.092543 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a392aa29-7d65-4306-a009-a10e05bd0ad5 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:33.098675 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:33.099428 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:33.099803 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:33.100083 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] PATH_INFO: `/v3/projects/a1d98123a2bf4cf09e26d057ff7bc316/users/e3c4b851c4504e76aaca747d1fa0fcfb/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:33.102455 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1d98123a2bf4cf09e26d057ff7bc316, user_id=e3c4b851c4504e76aaca747d1fa0fcfb, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:33.105853 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:33.118674 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1254/2519] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:33 2022] PUT /identity/v3/projects/a1d98123a2bf4cf09e26d057ff7bc316/users/e3c4b851c4504e76aaca747d1fa0fcfb/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:51:33.130385 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8e5e53-a21f-4213-9d9f-8d88d1f3f3b8 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:33.130743 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8e5e53-a21f-4213-9d9f-8d88d1f3f3b8 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:33.131113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8e5e53-a21f-4213-9d9f-8d88d1f3f3b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:33.166933 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-4b8e5e53-a21f-4213-9d9f-8d88d1f3f3b8 None None] MFA Rules not processed for user `e3c4b851c4504e76aaca747d1fa0fcfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:33.169315 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4b8e5e53-a21f-4213-9d9f-8d88d1f3f3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:33.256308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1266/2520] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:33 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:33.268647 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:33.270644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:33.343172 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96824898-ba8a-42ba-ac1a-9007269505ca admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:33.347574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:33.348240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:33.348521 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:33.348795 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:33.360836 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:33.362386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:33.375099 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1255/2521] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:33 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:33.572678 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8e5e53-a21f-4213-9d9f-8d88d1f3f3b8 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:33.584970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8e5e53-a21f-4213-9d9f-8d88d1f3f3b8 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:33.589764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38801092-6644-47ba-8426-86ac0b13c7bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:33.590637 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38801092-6644-47ba-8426-86ac0b13c7bb service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:33.590915 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38801092-6644-47ba-8426-86ac0b13c7bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:33.591187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38801092-6644-47ba-8426-86ac0b13c7bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:33.670916 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38801092-6644-47ba-8426-86ac0b13c7bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:33.672504 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38801092-6644-47ba-8426-86ac0b13c7bb service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:33.684567 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1267/2522] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:33 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:40.455057 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:40.456730 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:40.525480 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e093e73-e538-4ed1-93a3-ccf4b1e784ad service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:40.532077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ade63d0-ddd2-4e9c-b61c-93bc9415a381 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:40.533140 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade63d0-ddd2-4e9c-b61c-93bc9415a381 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:40.533688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade63d0-ddd2-4e9c-b61c-93bc9415a381 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:40.534192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade63d0-ddd2-4e9c-b61c-93bc9415a381 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:40.537169 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-7ade63d0-ddd2-4e9c-b61c-93bc9415a381 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:40.560135 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ade63d0-ddd2-4e9c-b61c-93bc9415a381 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:40.561688 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ade63d0-ddd2-4e9c-b61c-93bc9415a381 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:40.571821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1256/2523] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:40 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:42.633318 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38801092-6644-47ba-8426-86ac0b13c7bb service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:42.696010 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=100 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:42.715185 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38801092-6644-47ba-8426-86ac0b13c7bb service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:42.720738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:42.721507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:42.721935 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:42.722341 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] PATH_INFO: `/v3/users/4602605c0a6949a6b6833d46c9b7df15` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:42.723841 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4602605c0a6949a6b6833d46c9b7df15)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:42.725961 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:42.782712 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4602605c0a6949a6b6833d46c9b7df15'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:42.791545 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4602605c0a6949a6b6833d46c9b7df15'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:42.792970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4602605c0a6949a6b6833d46c9b7df15'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:42.793623 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4602605c0a6949a6b6833d46c9b7df15'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:42.803695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4602605c0a6949a6b6833d46c9b7df15'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:42.827307 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1268/2524] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:51:42 2022] DELETE /identity/v3/users/4602605c0a6949a6b6833d46c9b7df15 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:42.833358 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6671b7a-39b8-4eef-98d3-d8576f676752 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:42.834006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6671b7a-39b8-4eef-98d3-d8576f676752 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:42.834006 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6671b7a-39b8-4eef-98d3-d8576f676752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:42.857176 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-e6671b7a-39b8-4eef-98d3-d8576f676752 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:42.858444 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e6671b7a-39b8-4eef-98d3-d8576f676752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:42.929259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1257/2525] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:42.940683 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:42.942182 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:43.021884 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c56ea5-b7a4-47ed-aedd-efd8610f2cf0 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:43.026615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6ca7f5-5c88-4173-980f-29a4ed994c9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:43.027260 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6ca7f5-5c88-4173-980f-29a4ed994c9e service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:43.027636 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6ca7f5-5c88-4173-980f-29a4ed994c9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:43.027910 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6ca7f5-5c88-4173-980f-29a4ed994c9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:43.038463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6ca7f5-5c88-4173-980f-29a4ed994c9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:43.040471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6ca7f5-5c88-4173-980f-29a4ed994c9e service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:43.052981 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1269/2526] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:51:43.445967 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6671b7a-39b8-4eef-98d3-d8576f676752 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:43.448941 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e6671b7a-39b8-4eef-98d3-d8576f676752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:43.473500 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6671b7a-39b8-4eef-98d3-d8576f676752 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:43.479860 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:43.480719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:43.481113 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:43.481492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] PATH_INFO: `/v3/projects/70232ea3c5c443f9ac810901872889ed` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:43.483100 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=70232ea3c5c443f9ac810901872889ed)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:43.485708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:43.502266 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:43.555030 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '70232ea3c5c443f9ac810901872889ed', 'request_id': 'req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.562447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '70232ea3c5c443f9ac810901872889ed', 'request_id': 'req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.569186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '70232ea3c5c443f9ac810901872889ed', 'request_id': 'req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.585509 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] The token cache is being invalidate because project 70232ea3c5c443f9ac810901872889ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:51:43.585867 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.586507 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.588086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1258/2527] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:51:43 2022] DELETE /identity/v3/projects/70232ea3c5c443f9ac810901872889ed => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:51:43.592440 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6ca7f5-5c88-4173-980f-29a4ed994c9e service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:43.603366 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6ca7f5-5c88-4173-980f-29a4ed994c9e service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:43.684738 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:43.685448 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:43.685669 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:43.685947 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] PATH_INFO: `/v3/users/e3c4b851c4504e76aaca747d1fa0fcfb` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:43.687275 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3c4b851c4504e76aaca747d1fa0fcfb)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:43.689423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:43.703643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:43.748805 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3c4b851c4504e76aaca747d1fa0fcfb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.756833 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3c4b851c4504e76aaca747d1fa0fcfb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.757592 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3c4b851c4504e76aaca747d1fa0fcfb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.758213 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3c4b851c4504e76aaca747d1fa0fcfb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.767191 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-069f77ec-0592-4faa-9c7d-a72a80727140 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3c4b851c4504e76aaca747d1fa0fcfb'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:43.787685 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1270/2528] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:51:43 2022] DELETE /identity/v3/users/e3c4b851c4504e76aaca747d1fa0fcfb => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:44.113271 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:44.115402 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:44.136202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14df41d9-cdf5-4e82-a221-cc21c7d2efe3 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:44.209415 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:44.210070 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.210349 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.210659 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] PATH_INFO: `/v3/projects/a1d98123a2bf4cf09e26d057ff7bc316` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.211837 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1d98123a2bf4cf09e26d057ff7bc316)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:44.213778 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:44.271531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1d98123a2bf4cf09e26d057ff7bc316', 'request_id': 'req-ab057e99-5802-4c66-97a2-fa572dc0c9ea'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:44.279431 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1d98123a2bf4cf09e26d057ff7bc316', 'request_id': 'req-ab057e99-5802-4c66-97a2-fa572dc0c9ea'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:44.285825 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1d98123a2bf4cf09e26d057ff7bc316', 'request_id': 'req-ab057e99-5802-4c66-97a2-fa572dc0c9ea'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:44.301568 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] The token cache is being invalidate because project a1d98123a2bf4cf09e26d057ff7bc316 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:51:44.301864 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:44.302584 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ab057e99-5802-4c66-97a2-fa572dc0c9ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:44.304202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1259/2529] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:51:44 2022] DELETE /identity/v3/projects/a1d98123a2bf4cf09e26d057ff7bc316 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:44.309741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1400f7-ffdf-4058-87a5-9e72e743a365 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.310063 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1400f7-ffdf-4058-87a5-9e72e743a365 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.310257 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1400f7-ffdf-4058-87a5-9e72e743a365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.340001 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-2c1400f7-ffdf-4058-87a5-9e72e743a365 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:44.342352 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-2c1400f7-ffdf-4058-87a5-9e72e743a365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:44.427696 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1271/2530] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:51:44.436456 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fad827-4bea-42a3-ba83-3fb4c278f5d5 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.436754 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fad827-4bea-42a3-ba83-3fb4c278f5d5 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.437002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fad827-4bea-42a3-ba83-3fb4c278f5d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.462168 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-77fad827-4bea-42a3-ba83-3fb4c278f5d5 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:44.464654 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-77fad827-4bea-42a3-ba83-3fb4c278f5d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:44.478062 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1260/2531] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:51:44.484284 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1400f7-ffdf-4058-87a5-9e72e743a365 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:44.497534 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1400f7-ffdf-4058-87a5-9e72e743a365 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:44.505931 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:44.506982 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.507280 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.507714 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.508662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:51:44.509004 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:44.511894 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:44.526825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1272/2532] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:51:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:51:44.532880 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fad827-4bea-42a3-ba83-3fb4c278f5d5 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:44.549792 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fad827-4bea-42a3-ba83-3fb4c278f5d5 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:44.559618 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efc5a2b-3a1e-4153-a1c5-43e66eddc4e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:44.560691 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efc5a2b-3a1e-4153-a1c5-43e66eddc4e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.561202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efc5a2b-3a1e-4153-a1c5-43e66eddc4e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.561708 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efc5a2b-3a1e-4153-a1c5-43e66eddc4e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.563039 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efc5a2b-3a1e-4153-a1c5-43e66eddc4e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:44.567167 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efc5a2b-3a1e-4153-a1c5-43e66eddc4e3 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:44.618925 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1261/2533] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:51:44 2022] POST /identity/v3/projects => generated 385 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:44.624085 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:44.635760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266bfcf-9156-4bc1-8a56-81b2e02a2539 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:44.716102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a96a5af-234d-428e-bc06-09ef1b4c8363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:44.717160 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a96a5af-234d-428e-bc06-09ef1b4c8363 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.717652 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a96a5af-234d-428e-bc06-09ef1b4c8363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.718139 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a96a5af-234d-428e-bc06-09ef1b4c8363 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.719590 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a96a5af-234d-428e-bc06-09ef1b4c8363 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:44.723482 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a96a5af-234d-428e-bc06-09ef1b4c8363 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:44.760142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1273/2534] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:51:44 2022] POST /identity/v3/users => generated 293 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:51:44.764725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efc5a2b-3a1e-4153-a1c5-43e66eddc4e3 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:44.770720 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:44.779351 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efc5a2b-3a1e-4153-a1c5-43e66eddc4e3 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:44.788483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb6679b-3113-4690-bff0-39f06dc7ca9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:44.789510 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb6679b-3113-4690-bff0-39f06dc7ca9b admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.789979 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb6679b-3113-4690-bff0-39f06dc7ca9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.790445 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb6679b-3113-4690-bff0-39f06dc7ca9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.791766 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb6679b-3113-4690-bff0-39f06dc7ca9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:44.796315 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb6679b-3113-4690-bff0-39f06dc7ca9b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:44.815796 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1262/2535] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:44 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:51:44.820512 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a96a5af-234d-428e-bc06-09ef1b4c8363 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:44.828792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:44.833423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a96a5af-234d-428e-bc06-09ef1b4c8363 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:44.840372 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd880008-f655-4fdb-9672-f23ecdabc837 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:44.841561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd880008-f655-4fdb-9672-f23ecdabc837 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.841943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd880008-f655-4fdb-9672-f23ecdabc837 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.842759 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd880008-f655-4fdb-9672-f23ecdabc837 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.843309 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd880008-f655-4fdb-9672-f23ecdabc837 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:44.846897 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd880008-f655-4fdb-9672-f23ecdabc837 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:44.860094 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1274/2536] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:44 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:51:44.865504 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb6679b-3113-4690-bff0-39f06dc7ca9b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:44.878305 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb6679b-3113-4690-bff0-39f06dc7ca9b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:44.884517 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f456da4-9519-48e5-a2b9-379a82111475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:44.885759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f456da4-9519-48e5-a2b9-379a82111475 admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.886252 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f456da4-9519-48e5-a2b9-379a82111475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.886681 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f456da4-9519-48e5-a2b9-379a82111475 admin admin] PATH_INFO: `/v3/projects/2704ef762ed9499798074f3e871404b5/users/83d286ece9614dfdb8b70cd9c542da06/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.905366 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f456da4-9519-48e5-a2b9-379a82111475 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2704ef762ed9499798074f3e871404b5, user_id=83d286ece9614dfdb8b70cd9c542da06, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:44.908211 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f456da4-9519-48e5-a2b9-379a82111475 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:44.930770 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1263/2537] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:44 2022] PUT /identity/v3/projects/2704ef762ed9499798074f3e871404b5/users/83d286ece9614dfdb8b70cd9c542da06/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:44.942654 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb7383c-a250-45ee-a168-cac0664bf026 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:44.942952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb7383c-a250-45ee-a168-cac0664bf026 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:44.943230 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb7383c-a250-45ee-a168-cac0664bf026 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:44.985161 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9eb7383c-a250-45ee-a168-cac0664bf026 None None] MFA Rules not processed for user `83d286ece9614dfdb8b70cd9c542da06`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:44.987348 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9eb7383c-a250-45ee-a168-cac0664bf026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:45.096113 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1275/2538] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:45.102283 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f456da4-9519-48e5-a2b9-379a82111475 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:45.208587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f456da4-9519-48e5-a2b9-379a82111475 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:45.217652 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeba258-bd78-41b8-a25b-3686b593fb1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:45.219033 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeba258-bd78-41b8-a25b-3686b593fb1e admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:45.219551 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeba258-bd78-41b8-a25b-3686b593fb1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:45.220061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeba258-bd78-41b8-a25b-3686b593fb1e admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:45.221270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfeba258-bd78-41b8-a25b-3686b593fb1e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:45.225196 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfeba258-bd78-41b8-a25b-3686b593fb1e admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:45.270325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1264/2539] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:51:45 2022] POST /identity/v3/projects => generated 385 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:45.276903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb7383c-a250-45ee-a168-cac0664bf026 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:45.289764 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb7383c-a250-45ee-a168-cac0664bf026 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:45.379187 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db08af8c-4819-4b9e-96b1-77aea7060c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:45.380042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db08af8c-4819-4b9e-96b1-77aea7060c7f admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:45.380427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db08af8c-4819-4b9e-96b1-77aea7060c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:45.380874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db08af8c-4819-4b9e-96b1-77aea7060c7f admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:45.382055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db08af8c-4819-4b9e-96b1-77aea7060c7f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:45.386865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db08af8c-4819-4b9e-96b1-77aea7060c7f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:45.428273 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1276/2540] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:51:45 2022] POST /identity/v3/users => generated 293 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:45.435337 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeba258-bd78-41b8-a25b-3686b593fb1e admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:45.459111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeba258-bd78-41b8-a25b-3686b593fb1e admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:45.468243 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc274b2-a3a2-4711-8d69-d60063959688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:45.469526 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc274b2-a3a2-4711-8d69-d60063959688 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:45.470035 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc274b2-a3a2-4711-8d69-d60063959688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:45.470645 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc274b2-a3a2-4711-8d69-d60063959688 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:45.471980 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc274b2-a3a2-4711-8d69-d60063959688 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:45.475338 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc274b2-a3a2-4711-8d69-d60063959688 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:45.500060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1265/2541] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:45 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:51:45.502103 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db08af8c-4819-4b9e-96b1-77aea7060c7f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:45.518388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db08af8c-4819-4b9e-96b1-77aea7060c7f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:45.527226 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0ac030-9852-473e-b5c5-0dba20ecbb2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:45.528471 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0ac030-9852-473e-b5c5-0dba20ecbb2e admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:45.528922 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0ac030-9852-473e-b5c5-0dba20ecbb2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:45.529415 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0ac030-9852-473e-b5c5-0dba20ecbb2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:45.530665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0ac030-9852-473e-b5c5-0dba20ecbb2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:45.533605 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0ac030-9852-473e-b5c5-0dba20ecbb2e admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:45.558509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1277/2542] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:45 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:51:45.560740 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc274b2-a3a2-4711-8d69-d60063959688 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:45.584175 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc274b2-a3a2-4711-8d69-d60063959688 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:45.594595 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5442f8-fd33-4d01-a9ab-a16d6f62880a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:45.595968 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5442f8-fd33-4d01-a9ab-a16d6f62880a admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:45.596481 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5442f8-fd33-4d01-a9ab-a16d6f62880a admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:45.596982 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5442f8-fd33-4d01-a9ab-a16d6f62880a admin admin] PATH_INFO: `/v3/projects/ffc82faaf7924c9cb79f8cf2936bd82d/users/983a6c3ce60f4e498d4c10967533cd36/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:45.630119 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5442f8-fd33-4d01-a9ab-a16d6f62880a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffc82faaf7924c9cb79f8cf2936bd82d, user_id=983a6c3ce60f4e498d4c10967533cd36, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:45.637259 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5442f8-fd33-4d01-a9ab-a16d6f62880a admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:45.663406 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1266/2543] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:45 2022] PUT /identity/v3/projects/ffc82faaf7924c9cb79f8cf2936bd82d/users/983a6c3ce60f4e498d4c10967533cd36/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:45.765152 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df43bc4-5647-4439-a201-30b8b686398b None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:45.766036 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df43bc4-5647-4439-a201-30b8b686398b None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:45.766421 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df43bc4-5647-4439-a201-30b8b686398b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:45.853548 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-5df43bc4-5647-4439-a201-30b8b686398b None None] MFA Rules not processed for user `983a6c3ce60f4e498d4c10967533cd36`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:45.855819 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-5df43bc4-5647-4439-a201-30b8b686398b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:45.957561 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:45.984195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1278/2544] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:45.990711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5442f8-fd33-4d01-a9ab-a16d6f62880a admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:46.155676 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5442f8-fd33-4d01-a9ab-a16d6f62880a admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:46.163552 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb8b7ef-0c09-4cbf-8d10-91a8e296beb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:46.164684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb8b7ef-0c09-4cbf-8d10-91a8e296beb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:46.165368 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb8b7ef-0c09-4cbf-8d10-91a8e296beb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:46.165646 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb8b7ef-0c09-4cbf-8d10-91a8e296beb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:46.166336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb8b7ef-0c09-4cbf-8d10-91a8e296beb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:46.170241 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb8b7ef-0c09-4cbf-8d10-91a8e296beb0 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:46.226214 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1267/2545] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:51:45 2022] POST /identity/v3/projects => generated 381 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:51:46.233332 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df43bc4-5647-4439-a201-30b8b686398b None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:46.255404 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df43bc4-5647-4439-a201-30b8b686398b None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:46.353295 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c5743c-1d8d-438d-9228-c0f43c717f34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:46.355023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5743c-1d8d-438d-9228-c0f43c717f34 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:46.355571 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5743c-1d8d-438d-9228-c0f43c717f34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:46.355984 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5743c-1d8d-438d-9228-c0f43c717f34 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:46.356822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c5743c-1d8d-438d-9228-c0f43c717f34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:46.358991 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c5743c-1d8d-438d-9228-c0f43c717f34 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:46.390566 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1279/2546] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:51:46 2022] POST /identity/v3/users => generated 297 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:46.396669 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb8b7ef-0c09-4cbf-8d10-91a8e296beb0 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:46.409313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:46.417725 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb8b7ef-0c09-4cbf-8d10-91a8e296beb0 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:46.428060 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a4c5f-192b-4e96-b68b-2c4f11a137b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:46.428875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a4c5f-192b-4e96-b68b-2c4f11a137b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:46.429260 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a4c5f-192b-4e96-b68b-2c4f11a137b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:46.429638 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a4c5f-192b-4e96-b68b-2c4f11a137b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:46.430447 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a4c5f-192b-4e96-b68b-2c4f11a137b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:46.432845 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a4c5f-192b-4e96-b68b-2c4f11a137b5 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:46.454202 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1268/2547] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:46 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:51:46.461249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c5743c-1d8d-438d-9228-c0f43c717f34 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:46.484430 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c5743c-1d8d-438d-9228-c0f43c717f34 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:46.494023 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccab0f1c-9df9-43f3-b756-970aecae727f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:46.496055 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab0f1c-9df9-43f3-b756-970aecae727f admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:46.496538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab0f1c-9df9-43f3-b756-970aecae727f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:46.497017 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab0f1c-9df9-43f3-b756-970aecae727f admin admin] PATH_INFO: `/v3/projects/263ab15899ad48599c2e3a007a54781b/users/f75de18dcf6c4f4cb108272256894072/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:46.526121 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccab0f1c-9df9-43f3-b756-970aecae727f admin admin] RBAC: Authorizing `identity:create_grant(project_id=263ab15899ad48599c2e3a007a54781b, user_id=f75de18dcf6c4f4cb108272256894072, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:46.531827 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccab0f1c-9df9-43f3-b756-970aecae727f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:46.559551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1280/2548] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:46 2022] PUT /identity/v3/projects/263ab15899ad48599c2e3a007a54781b/users/f75de18dcf6c4f4cb108272256894072/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:46.565262 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a4c5f-192b-4e96-b68b-2c4f11a137b5 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:46.577321 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a4c5f-192b-4e96-b68b-2c4f11a137b5 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:46.661071 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e0d507-8b6b-4dae-b23d-441b636a59d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:46.661750 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e0d507-8b6b-4dae-b23d-441b636a59d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:46.662121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e0d507-8b6b-4dae-b23d-441b636a59d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:46.662604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e0d507-8b6b-4dae-b23d-441b636a59d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:46.663476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e0d507-8b6b-4dae-b23d-441b636a59d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:46.665458 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e0d507-8b6b-4dae-b23d-441b636a59d0 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:46.679105 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1269/2549] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:46 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:51:46.684245 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccab0f1c-9df9-43f3-b756-970aecae727f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:46.696218 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccab0f1c-9df9-43f3-b756-970aecae727f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:46.706339 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:46.707535 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] REQUEST_METHOD: `PUT` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:46.708024 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:46.708510 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] PATH_INFO: `/v3/projects/263ab15899ad48599c2e3a007a54781b/users/f75de18dcf6c4f4cb108272256894072/roles/a68d428c5b8c41d69e3c1432c46d2dff` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:46.712211 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=263ab15899ad48599c2e3a007a54781b, user_id=f75de18dcf6c4f4cb108272256894072, role_id=a68d428c5b8c41d69e3c1432c46d2dff)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:46.718096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:46.738925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1281/2550] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:46 2022] PUT /identity/v3/projects/263ab15899ad48599c2e3a007a54781b/users/f75de18dcf6c4f4cb108272256894072/roles/a68d428c5b8c41d69e3c1432c46d2dff => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:46.755328 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba93f8-b843-4dfa-bf5b-75077749baca None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:46.755706 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba93f8-b843-4dfa-bf5b-75077749baca None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:46.756052 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba93f8-b843-4dfa-bf5b-75077749baca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:46.796348 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-6cba93f8-b843-4dfa-bf5b-75077749baca None None] MFA Rules not processed for user `f75de18dcf6c4f4cb108272256894072`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:46.797873 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6cba93f8-b843-4dfa-bf5b-75077749baca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:46.908916 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1270/2551] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:46 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:46.926265 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:46.928689 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:47.017943 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8590d0-06cf-4ef9-b39f-14d6c1dd1e0f admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:47.022667 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9d9c2-4267-477d-9ac3-bbf5a47074e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:47.023463 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9d9c2-4267-477d-9ac3-bbf5a47074e6 service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:47.023850 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9d9c2-4267-477d-9ac3-bbf5a47074e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:47.024228 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9d9c2-4267-477d-9ac3-bbf5a47074e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:47.111749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd9d9c2-4267-477d-9ac3-bbf5a47074e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:47.114180 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd9d9c2-4267-477d-9ac3-bbf5a47074e6 service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:47.119555 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:47.129192 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1282/2552] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:46 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:48.952707 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba93f8-b843-4dfa-bf5b-75077749baca None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:48.955192 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-6cba93f8-b843-4dfa-bf5b-75077749baca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:48.974714 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba93f8-b843-4dfa-bf5b-75077749baca None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:48.979978 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c35981-86a1-4984-8966-029519565083 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:48.982115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c35981-86a1-4984-8966-029519565083 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:48.984524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c35981-86a1-4984-8966-029519565083 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:48.985111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c35981-86a1-4984-8966-029519565083 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:49.087424 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c35981-86a1-4984-8966-029519565083 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:49.089624 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c35981-86a1-4984-8966-029519565083 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:49.100680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1271/2553] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:48 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:51:49.894891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9d9c2-4267-477d-9ac3-bbf5a47074e6 service cinder] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:49.904645 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9d9c2-4267-477d-9ac3-bbf5a47074e6 service cinder] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:49.908484 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa0c33a-f30f-479b-8166-fe88e47d13de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:49.909107 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0c33a-f30f-479b-8166-fe88e47d13de service cinder] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:49.909388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0c33a-f30f-479b-8166-fe88e47d13de service cinder] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:49.909662 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0c33a-f30f-479b-8166-fe88e47d13de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:49.920668 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa0c33a-f30f-479b-8166-fe88e47d13de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:49.922151 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa0c33a-f30f-479b-8166-fe88e47d13de service cinder] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:49.933852 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1283/2554] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:49 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:51:54.117639 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c35981-86a1-4984-8966-029519565083 service cinder] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:54.143112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:54.239329 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c35981-86a1-4984-8966-029519565083 service cinder] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:54.245411 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:54.246069 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:54.246345 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:54.246658 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] PATH_INFO: `/v3/users/83d286ece9614dfdb8b70cd9c542da06` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:54.247944 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=83d286ece9614dfdb8b70cd9c542da06)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:54.249998 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:54.314548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '83d286ece9614dfdb8b70cd9c542da06'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:54.324861 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '83d286ece9614dfdb8b70cd9c542da06'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:54.325819 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '83d286ece9614dfdb8b70cd9c542da06'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:54.326541 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '83d286ece9614dfdb8b70cd9c542da06'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:54.337361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '83d286ece9614dfdb8b70cd9c542da06'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:54.363831 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1272/2555] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:51:54 2022] DELETE /identity/v3/users/83d286ece9614dfdb8b70cd9c542da06 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:54.368154 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53dbf6-ffd9-4436-ac01-fe5741dbedd9 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:54.369003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53dbf6-ffd9-4436-ac01-fe5741dbedd9 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:54.369003 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53dbf6-ffd9-4436-ac01-fe5741dbedd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:54.401468 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-ac53dbf6-ffd9-4436-ac01-fe5741dbedd9 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:54.403710 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ac53dbf6-ffd9-4436-ac01-fe5741dbedd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:54.484857 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1284/2556] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:51:54.496561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:54.497542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:54.605990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a5c12-11b3-441c-addc-3dcec4737dcb admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:54.610405 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:54.611465 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:54.611821 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:54.612163 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:54.624560 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:54.626043 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:54.639817 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1273/2557] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:54.839249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac53dbf6-ffd9-4436-ac01-fe5741dbedd9 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:54.842909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-ac53dbf6-ffd9-4436-ac01-fe5741dbedd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:55.034116 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac53dbf6-ffd9-4436-ac01-fe5741dbedd9 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:55.044726 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '41009aec3b1843aa80323c374cc01461', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:55.045789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:55.046271 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:55.046789 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:55.050394 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:55.077413 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:55.079434 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:55.080673 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:55.082144 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:55.102470 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e8ee02-ac62-404e-857a-59b7b918668a service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:55.111715 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:55.113692 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:55.114096 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:55.114478 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] PATH_INFO: `/v3/projects/2704ef762ed9499798074f3e871404b5` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:55.116786 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2704ef762ed9499798074f3e871404b5)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:55.119061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:55.120043 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1285/2558] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:51:55.171396 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:55.201306 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2704ef762ed9499798074f3e871404b5', 'request_id': 'req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.209718 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2704ef762ed9499798074f3e871404b5', 'request_id': 'req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.226684 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2704ef762ed9499798074f3e871404b5', 'request_id': 'req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.250767 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] The token cache is being invalidate because project 2704ef762ed9499798074f3e871404b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:51:55.251269 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.252143 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.258289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1274/2559] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:51:55 2022] DELETE /identity/v3/projects/2704ef762ed9499798074f3e871404b5 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:55.260615 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:55.273285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc3a8ab-84dd-4db3-8d72-c4168f64f9f9 service nova] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:55.364446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:55.365749 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:55.366360 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:55.367042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] PATH_INFO: `/v3/users/983a6c3ce60f4e498d4c10967533cd36` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:55.369978 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] RBAC: Authorizing `identity:delete_user(user_id=983a6c3ce60f4e498d4c10967533cd36)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:55.372677 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:55.399547 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:55.483970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '983a6c3ce60f4e498d4c10967533cd36'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.493895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '983a6c3ce60f4e498d4c10967533cd36'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.495129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '983a6c3ce60f4e498d4c10967533cd36'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.496140 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '983a6c3ce60f4e498d4c10967533cd36'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.509595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '983a6c3ce60f4e498d4c10967533cd36'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:55.539591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1286/2560] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:51:55 2022] DELETE /identity/v3/users/983a6c3ce60f4e498d4c10967533cd36 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:51:55.956126 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:55.958627 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:55.988313 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22691bb-54ec-44fb-aaa0-a95e84bfd05b admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:56.076186 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:56.077370 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:56.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:56.078364 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] PATH_INFO: `/v3/projects/ffc82faaf7924c9cb79f8cf2936bd82d` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:56.080693 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffc82faaf7924c9cb79f8cf2936bd82d)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:56.084538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:56.179946 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffc82faaf7924c9cb79f8cf2936bd82d', 'request_id': 'req-e86ca333-6472-4b44-aa93-aaa07789e11f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.194940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffc82faaf7924c9cb79f8cf2936bd82d', 'request_id': 'req-e86ca333-6472-4b44-aa93-aaa07789e11f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.209606 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffc82faaf7924c9cb79f8cf2936bd82d', 'request_id': 'req-e86ca333-6472-4b44-aa93-aaa07789e11f'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.238292 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] The token cache is being invalidate because project ffc82faaf7924c9cb79f8cf2936bd82d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:51:56.238863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.239993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.242587 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1275/2561] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:51:55 2022] DELETE /identity/v3/projects/ffc82faaf7924c9cb79f8cf2936bd82d => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:51:56.248783 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:56.251308 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:56.276607 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d574883-2fd7-4dc3-b8b1-9a4c5ef18142 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:56.361509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:56.362695 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:56.363184 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:56.363672 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] PATH_INFO: `/v3/users/f75de18dcf6c4f4cb108272256894072` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:56.365758 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f75de18dcf6c4f4cb108272256894072)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:56.369681 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:56.453338 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f75de18dcf6c4f4cb108272256894072'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.466800 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f75de18dcf6c4f4cb108272256894072'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.467975 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f75de18dcf6c4f4cb108272256894072'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.469231 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f75de18dcf6c4f4cb108272256894072'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.481523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-96f5b257-46f7-41c7-b82a-51a92dfd24d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f75de18dcf6c4f4cb108272256894072'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:56.518505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1287/2562] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:51:56 2022] DELETE /identity/v3/users/f75de18dcf6c4f4cb108272256894072 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:57.294108 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:57.297977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:57.327531 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86ca333-6472-4b44-aa93-aaa07789e11f admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:57.399737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:57.421354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:57.422863 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:57.423146 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:57.423460 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] PATH_INFO: `/v3/projects/263ab15899ad48599c2e3a007a54781b` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:57.424741 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=263ab15899ad48599c2e3a007a54781b)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:57.426759 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:57.493580 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '263ab15899ad48599c2e3a007a54781b', 'request_id': 'req-6b68bb89-715f-4822-a1b6-5a04d465ff0b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:57.501829 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '263ab15899ad48599c2e3a007a54781b', 'request_id': 'req-6b68bb89-715f-4822-a1b6-5a04d465ff0b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:57.510783 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '263ab15899ad48599c2e3a007a54781b', 'request_id': 'req-6b68bb89-715f-4822-a1b6-5a04d465ff0b'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:57.530513 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] The token cache is being invalidate because project 263ab15899ad48599c2e3a007a54781b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:51:57.531136 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:57.531990 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6b68bb89-715f-4822-a1b6-5a04d465ff0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:57.534270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1276/2563] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:51:57 2022] DELETE /identity/v3/projects/263ab15899ad48599c2e3a007a54781b => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:57.539917 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065aa63c-0bc2-4232-9502-3c0f1b37c805 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:57.540760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065aa63c-0bc2-4232-9502-3c0f1b37c805 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:57.540760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065aa63c-0bc2-4232-9502-3c0f1b37c805 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:57.567890 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-065aa63c-0bc2-4232-9502-3c0f1b37c805 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:57.568973 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-065aa63c-0bc2-4232-9502-3c0f1b37c805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:57.576148 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:57.656766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1288/2564] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:57.665958 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3997405c-f73c-4bd9-b723-45270135b836 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:57.666874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3997405c-f73c-4bd9-b723-45270135b836 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:57.667441 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3997405c-f73c-4bd9-b723-45270135b836 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:57.693274 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-3997405c-f73c-4bd9-b723-45270135b836 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:57.695278 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3997405c-f73c-4bd9-b723-45270135b836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:57.712090 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1277/2565] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:51:57.718620 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065aa63c-0bc2-4232-9502-3c0f1b37c805 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:57.740246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065aa63c-0bc2-4232-9502-3c0f1b37c805 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:57.748132 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:57.749100 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:57.749688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:57.750178 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:57.751141 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:51:57.751792 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:57.754441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:57.774870 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1289/2566] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:51:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:51:57.780973 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3997405c-f73c-4bd9-b723-45270135b836 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:57.794991 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3997405c-f73c-4bd9-b723-45270135b836 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:57.801042 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59704fb-42b6-4e9e-8e4f-7e204ab96b7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:57.801883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59704fb-42b6-4e9e-8e4f-7e204ab96b7d admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:57.802574 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59704fb-42b6-4e9e-8e4f-7e204ab96b7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:57.803073 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59704fb-42b6-4e9e-8e4f-7e204ab96b7d admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:57.804285 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59704fb-42b6-4e9e-8e4f-7e204ab96b7d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:57.808738 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59704fb-42b6-4e9e-8e4f-7e204ab96b7d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:57.867002 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1278/2567] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:51:57 2022] POST /identity/v3/projects => generated 371 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:51:57.870569 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:57.889575 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d127640-319f-42fb-91f2-d645f6e19905 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:58.006612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993b112-0142-4373-b58d-482d9ba705fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:58.007255 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993b112-0142-4373-b58d-482d9ba705fc admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:58.007540 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993b112-0142-4373-b58d-482d9ba705fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:58.007820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993b112-0142-4373-b58d-482d9ba705fc admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:58.008473 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4993b112-0142-4373-b58d-482d9ba705fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:58.011006 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4993b112-0142-4373-b58d-482d9ba705fc admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:58.053109 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1290/2568] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:51:57 2022] POST /identity/v3/users => generated 286 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:51:58.060289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59704fb-42b6-4e9e-8e4f-7e204ab96b7d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:58.077798 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59704fb-42b6-4e9e-8e4f-7e204ab96b7d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:58.086041 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2429b7-d066-4631-a749-741659d0b74c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:58.087128 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2429b7-d066-4631-a749-741659d0b74c admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:58.087623 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2429b7-d066-4631-a749-741659d0b74c admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:58.088682 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2429b7-d066-4631-a749-741659d0b74c admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:58.089789 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2429b7-d066-4631-a749-741659d0b74c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:58.092812 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d2429b7-d066-4631-a749-741659d0b74c admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:58.106231 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1279/2569] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:58 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:51:58.110976 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993b112-0142-4373-b58d-482d9ba705fc admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:58.123096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993b112-0142-4373-b58d-482d9ba705fc admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:58.129830 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f849941-b779-4d77-b40e-357daf6a9601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:58.130066 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f849941-b779-4d77-b40e-357daf6a9601 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:58.130621 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f849941-b779-4d77-b40e-357daf6a9601 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:58.130967 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f849941-b779-4d77-b40e-357daf6a9601 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:58.131741 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f849941-b779-4d77-b40e-357daf6a9601 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:58.133506 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f849941-b779-4d77-b40e-357daf6a9601 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:58.149998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1291/2570] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:51:58 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:51:58.156666 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2429b7-d066-4631-a749-741659d0b74c admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:58.168420 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2429b7-d066-4631-a749-741659d0b74c admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:58.173791 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:58.174477 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:58.174801 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:58.175079 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] PATH_INFO: `/v3/projects/4f1bcb9d3eda4d74bbd1a9225417684e/users/25cf656c55324c9b847ffd3c26f73d63/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:58.201719 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f1bcb9d3eda4d74bbd1a9225417684e, user_id=25cf656c55324c9b847ffd3c26f73d63, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:58.207518 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:58.234020 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1280/2571] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:51:58 2022] PUT /identity/v3/projects/4f1bcb9d3eda4d74bbd1a9225417684e/users/25cf656c55324c9b847ffd3c26f73d63/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:58.246776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c06ee-5dae-4bd1-9214-cabee6630d0f None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:58.246776 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c06ee-5dae-4bd1-9214-cabee6630d0f None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:58.247128 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c06ee-5dae-4bd1-9214-cabee6630d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:58.287382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-9f3c06ee-5dae-4bd1-9214-cabee6630d0f None None] MFA Rules not processed for user `25cf656c55324c9b847ffd3c26f73d63`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:58.288821 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-9f3c06ee-5dae-4bd1-9214-cabee6630d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:58.382382 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1292/2572] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:58 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:51:58.395572 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:58.397690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:58.477499 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d3485a-53c5-4325-a39c-d7cd7f12b86d admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:58.483952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b719d545-84e9-4836-8c6a-14579d7514f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:58.484739 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b719d545-84e9-4836-8c6a-14579d7514f5 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:58.485124 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b719d545-84e9-4836-8c6a-14579d7514f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:58.485488 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b719d545-84e9-4836-8c6a-14579d7514f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:58.496716 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b719d545-84e9-4836-8c6a-14579d7514f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:58.498114 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b719d545-84e9-4836-8c6a-14579d7514f5 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:58.509289 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1281/2573] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:58 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:51:58.552908 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3c06ee-5dae-4bd1-9214-cabee6630d0f None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:58.564544 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:58.695234 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3c06ee-5dae-4bd1-9214-cabee6630d0f None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:58.703328 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:58.704453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:58.704970 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:58.705460 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] PATH_INFO: `/v3/users/25cf656c55324c9b847ffd3c26f73d63` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:58.707718 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25cf656c55324c9b847ffd3c26f73d63)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:58.711656 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:58.784102 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25cf656c55324c9b847ffd3c26f73d63'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:58.793388 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25cf656c55324c9b847ffd3c26f73d63'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:58.794290 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25cf656c55324c9b847ffd3c26f73d63'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:58.795249 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25cf656c55324c9b847ffd3c26f73d63'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:58.806774 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25cf656c55324c9b847ffd3c26f73d63'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:58.834997 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1293/2574] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:51:58 2022] DELETE /identity/v3/users/25cf656c55324c9b847ffd3c26f73d63 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:58.840484 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebf187f-ed97-46d1-b9a6-936fc5d04279 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:58.840907 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebf187f-ed97-46d1-b9a6-936fc5d04279 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:58.841275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebf187f-ed97-46d1-b9a6-936fc5d04279 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:58.863690 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-bebf187f-ed97-46d1-b9a6-936fc5d04279 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:58.864952 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bebf187f-ed97-46d1-b9a6-936fc5d04279 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:58.876011 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:58.941734 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1282/2575] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:51:58.953387 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:58.955453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:59.024998 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67fcd9c-8197-413c-8464-41353bbad1e6 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:59.029532 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ff1f0-700e-4c4d-ab3e-d7e1e8a1aac0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:59.030168 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ff1f0-700e-4c4d-ab3e-d7e1e8a1aac0 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:59.030446 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ff1f0-700e-4c4d-ab3e-d7e1e8a1aac0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:59.030757 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ff1f0-700e-4c4d-ab3e-d7e1e8a1aac0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:59.044035 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ff1f0-700e-4c4d-ab3e-d7e1e8a1aac0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:59.046441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ff1f0-700e-4c4d-ab3e-d7e1e8a1aac0 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:59.064771 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1294/2576] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:51:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:51:59.551270 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebf187f-ed97-46d1-b9a6-936fc5d04279 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:59.553325 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-bebf187f-ed97-46d1-b9a6-936fc5d04279 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:59.568361 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebf187f-ed97-46d1-b9a6-936fc5d04279 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:59.575483 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:59.576354 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:59.576646 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:59.576953 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] PATH_INFO: `/v3/projects/4f1bcb9d3eda4d74bbd1a9225417684e` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:59.580111 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f1bcb9d3eda4d74bbd1a9225417684e)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:59.582077 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:59.640970 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f1bcb9d3eda4d74bbd1a9225417684e', 'request_id': 'req-6e69ca8a-0a04-4367-bd54-94e5698638a0'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:59.648103 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f1bcb9d3eda4d74bbd1a9225417684e', 'request_id': 'req-6e69ca8a-0a04-4367-bd54-94e5698638a0'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:59.654689 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f1bcb9d3eda4d74bbd1a9225417684e', 'request_id': 'req-6e69ca8a-0a04-4367-bd54-94e5698638a0'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:59.669851 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] The token cache is being invalidate because project 4f1bcb9d3eda4d74bbd1a9225417684e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:51:59.670206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:59.671046 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-6e69ca8a-0a04-4367-bd54-94e5698638a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:51:59.672977 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1283/2577] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:51:59 2022] DELETE /identity/v3/projects/4f1bcb9d3eda4d74bbd1a9225417684e => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:51:59.678665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cefb8be-af65-4e72-859c-e3be7d188083 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:59.679552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cefb8be-af65-4e72-859c-e3be7d188083 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:59.679766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cefb8be-af65-4e72-859c-e3be7d188083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:59.703096 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-4cefb8be-af65-4e72-859c-e3be7d188083 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:59.704503 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-4cefb8be-af65-4e72-859c-e3be7d188083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:59.729930 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:51:59.782453 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1295/2578] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:51:59.791527 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37828df-039c-4cd2-a6b0-e11abb6dd536 None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:59.792577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37828df-039c-4cd2-a6b0-e11abb6dd536 None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:59.792577 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37828df-039c-4cd2-a6b0-e11abb6dd536 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:59.816540 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-c37828df-039c-4cd2-a6b0-e11abb6dd536 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:51:59.818009 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c37828df-039c-4cd2-a6b0-e11abb6dd536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:51:59.829326 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1284/2579] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:51:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:51:59.840092 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cefb8be-af65-4e72-859c-e3be7d188083 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:59.850877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cefb8be-af65-4e72-859c-e3be7d188083 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:59.856909 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:59.857538 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:59.857825 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:59.858074 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] PATH_INFO: `/v3/domains` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:59.859027 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81811) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 00:51:59.859427 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:59.861402 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:59.876735 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1296/2580] 10.222.0.26 () {58 vars in 1153 bytes} [Wed Oct 19 00:51:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 00:51:59.882010 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37828df-039c-4cd2-a6b0-e11abb6dd536 None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:59.893760 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37828df-039c-4cd2-a6b0-e11abb6dd536 None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:51:59.899544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a70b89-5967-4ce0-811c-f38194070c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:51:59.900230 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a70b89-5967-4ce0-811c-f38194070c11 admin admin] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:51:59.900553 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a70b89-5967-4ce0-811c-f38194070c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:51:59.900975 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a70b89-5967-4ce0-811c-f38194070c11 admin admin] PATH_INFO: `/v3/projects` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:51:59.901940 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a70b89-5967-4ce0-811c-f38194070c11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:51:59.904121 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a70b89-5967-4ce0-811c-f38194070c11 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:51:59.946895 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1285/2581] 10.222.0.26 () {60 vars in 1153 bytes} [Wed Oct 19 00:51:59 2022] POST /identity/v3/projects => generated 373 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 00:51:59.951995 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:51:59.966665 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7690cb86-9148-4f37-b844-a8b47158120a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:00.060334 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8998948-6e2d-4252-bd9e-005205e93c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:00.060968 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8998948-6e2d-4252-bd9e-005205e93c32 admin admin] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:00.061246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8998948-6e2d-4252-bd9e-005205e93c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:00.061518 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8998948-6e2d-4252-bd9e-005205e93c32 admin admin] PATH_INFO: `/v3/users` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:00.062150 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8998948-6e2d-4252-bd9e-005205e93c32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:00.064195 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8998948-6e2d-4252-bd9e-005205e93c32 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:00.091134 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1297/2582] 10.222.0.26 () {60 vars in 1144 bytes} [Wed Oct 19 00:51:59 2022] POST /identity/v3/users => generated 287 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 00:52:00.095862 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a70b89-5967-4ce0-811c-f38194070c11 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:00.109171 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a70b89-5967-4ce0-811c-f38194070c11 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:00.115206 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33733112-cd02-417b-931a-0b7f51476a83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:00.115888 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33733112-cd02-417b-931a-0b7f51476a83 admin admin] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:00.116208 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33733112-cd02-417b-931a-0b7f51476a83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:00.116548 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33733112-cd02-417b-931a-0b7f51476a83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:00.117253 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33733112-cd02-417b-931a-0b7f51476a83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:00.119296 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33733112-cd02-417b-931a-0b7f51476a83 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:00.132917 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1286/2583] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:52:00 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 00:52:00.137648 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8998948-6e2d-4252-bd9e-005205e93c32 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:00.150822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8998948-6e2d-4252-bd9e-005205e93c32 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:00.156816 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7c4ad7-e087-4cef-a876-bd06e5391346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:00.157523 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c4ad7-e087-4cef-a876-bd06e5391346 admin admin] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:00.157885 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c4ad7-e087-4cef-a876-bd06e5391346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:00.158241 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c4ad7-e087-4cef-a876-bd06e5391346 admin admin] PATH_INFO: `/v3/roles` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:00.159122 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7c4ad7-e087-4cef-a876-bd06e5391346 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:00.161256 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7c4ad7-e087-4cef-a876-bd06e5391346 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:00.176898 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1298/2584] 10.222.0.26 () {58 vars in 1122 bytes} [Wed Oct 19 00:52:00 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 00:52:00.182629 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33733112-cd02-417b-931a-0b7f51476a83 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:00.193875 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33733112-cd02-417b-931a-0b7f51476a83 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:00.200746 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:00.201793 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:00.202160 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:00.202604 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] PATH_INFO: `/v3/projects/66f8ce6f763241338027d186629b47ce/users/6d8443dea9434f67b21c3b2b8aa2e72f/roles/d729ee72a6d746959af49c5cab72ddd6` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:00.224086 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=66f8ce6f763241338027d186629b47ce, user_id=6d8443dea9434f67b21c3b2b8aa2e72f, role_id=d729ee72a6d746959af49c5cab72ddd6)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:00.228177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:00.249178 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1287/2585] 10.222.0.26 () {60 vars in 1483 bytes} [Wed Oct 19 00:52:00 2022] PUT /identity/v3/projects/66f8ce6f763241338027d186629b47ce/users/6d8443dea9434f67b21c3b2b8aa2e72f/roles/d729ee72a6d746959af49c5cab72ddd6 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:00.263246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770e3bd0-3d26-4f0c-aeb4-69c303149c8e None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:00.263246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770e3bd0-3d26-4f0c-aeb4-69c303149c8e None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:00.263246 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770e3bd0-3d26-4f0c-aeb4-69c303149c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:00.305697 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-770e3bd0-3d26-4f0c-aeb4-69c303149c8e None None] MFA Rules not processed for user `6d8443dea9434f67b21c3b2b8aa2e72f`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:52:00.307946 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-770e3bd0-3d26-4f0c-aeb4-69c303149c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:00.412042 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1299/2586] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:52:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:52:00.424395 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:00.426371 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:00.501044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:00.536386 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e317c5f7-2dc1-4852-b2f7-0523a52959ae admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:00.542710 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66cf953-6c09-41b5-bce8-5b8a9932b936 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '13270581a86e430886c878ff84302764', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:00.543744 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66cf953-6c09-41b5-bce8-5b8a9932b936 service cinder] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:00.544229 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66cf953-6c09-41b5-bce8-5b8a9932b936 service cinder] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:00.544722 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66cf953-6c09-41b5-bce8-5b8a9932b936 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:00.570082 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66cf953-6c09-41b5-bce8-5b8a9932b936 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:00.573813 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66cf953-6c09-41b5-bce8-5b8a9932b936 service cinder] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:00.592476 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1288/2587] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:52:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:52:02.630849 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770e3bd0-3d26-4f0c-aeb4-69c303149c8e None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:02.633900 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-770e3bd0-3d26-4f0c-aeb4-69c303149c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:02.691851 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:02.760582 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770e3bd0-3d26-4f0c-aeb4-69c303149c8e None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:02.765412 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:02.766142 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:02.766441 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:02.766736 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:02.771591 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:02.907464 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:02.910270 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:02.930230 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1300/2588] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:52:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:52:10.583607 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=10 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:12.922013 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=40 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:13.734112 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec06a28-05b4-439a-8d9a-4cfcab77062e None None] REQUEST_METHOD: `POST` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:13.735582 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec06a28-05b4-439a-8d9a-4cfcab77062e None None] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:13.736307 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec06a28-05b4-439a-8d9a-4cfcab77062e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:13.751877 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:13.765379 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e32836-0009-447c-ba08-f2925dbdc1f9 service neutron] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:13.771056 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:13.771720 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:13.772007 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:13.772289 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] PATH_INFO: `/v3/users/6d8443dea9434f67b21c3b2b8aa2e72f` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:13.773845 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d8443dea9434f67b21c3b2b8aa2e72f)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:13.777149 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:13.860538 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.auth.core [None req-0ec06a28-05b4-439a-8d9a-4cfcab77062e None None] MFA Rules not processed for user `f7fcaf24d2ff423a90d9ea6747a67a05`. Rule list: `[]` (Enabled: `True`). {{(pid=81812) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:52:13.863050 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0ec06a28-05b4-439a-8d9a-4cfcab77062e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:13.876301 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d8443dea9434f67b21c3b2b8aa2e72f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:13.904095 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d8443dea9434f67b21c3b2b8aa2e72f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:13.904891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d8443dea9434f67b21c3b2b8aa2e72f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:13.906631 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d8443dea9434f67b21c3b2b8aa2e72f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:13.919658 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-016abc3c-1e61-4d74-b055-317f23421716 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d8443dea9434f67b21c3b2b8aa2e72f'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:13.956953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1301/2589] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:52:13 2022] DELETE /identity/v3/users/6d8443dea9434f67b21c3b2b8aa2e72f => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:13.968786 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd24a7a-076e-493c-af2c-9ca2772ec235 None None] REQUEST_METHOD: `POST` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:13.969285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd24a7a-076e-493c-af2c-9ca2772ec235 None None] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:13.969777 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd24a7a-076e-493c-af2c-9ca2772ec235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:14.009780 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.auth.core [None req-3bd24a7a-076e-493c-af2c-9ca2772ec235 None None] MFA Rules not processed for user `ded9b81a3e894fd88539e86bbeb17ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81811) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 00:52:14.012811 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3bd24a7a-076e-493c-af2c-9ca2772ec235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:14.019247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1289/2590] 10.222.0.26 () {58 vars in 967 bytes} [Wed Oct 19 00:52:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 00:52:14.083573 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec06a28-05b4-439a-8d9a-4cfcab77062e None None] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:14.085336 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-0ec06a28-05b4-439a-8d9a-4cfcab77062e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:14.178717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1302/2591] 10.222.0.26 () {58 vars in 958 bytes} [Wed Oct 19 00:52:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 00:52:14.180802 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd24a7a-076e-493c-af2c-9ca2772ec235 None None] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:14.184717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-3bd24a7a-076e-493c-af2c-9ca2772ec235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:14.232174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec06a28-05b4-439a-8d9a-4cfcab77062e None None] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:14.320275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:14.320912 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:14.321193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:14.321464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:14.324672 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:14.340215 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:14.341874 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:14.356506 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1290/2592] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:52:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 00:52:14.357785 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:14.360061 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:14.364813 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:14.367537 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=100 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:14.407766 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd24a7a-076e-493c-af2c-9ca2772ec235 None None] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:14.415752 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '33b9c777f8284edeb758b00de044fd45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:14.416401 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:14.416688 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:14.416955 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:14.472098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e672f87-aa4b-483a-b2f1-8f8e04886717 service placement] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:14.476363 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5756f4ff604a413b8af1f12c82cc4870', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:14.476995 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] REQUEST_METHOD: `GET` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:14.477280 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:14.477554 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:14.488834 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:14.490346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:14.491903 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:14.494560 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:14.503686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1291/2593] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:52:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:52:14.504576 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:14.507895 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1303/2594] 10.222.0.26 () {60 vars in 1384 bytes} [Wed Oct 19 00:52:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 00:52:14.508240 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:14.526737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a67d2-9c40-41c9-ab5f-32a19bf38500 service neutron] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:14.532993 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:14.533964 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:14.534346 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:14.534752 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] PATH_INFO: `/v3/users/f738e67d8c2340028da1254b60d7810e` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:14.536195 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=f738e67d8c2340028da1254b60d7810e)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:14.538284 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:14.651193 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f738e67d8c2340028da1254b60d7810e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:14.664561 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f738e67d8c2340028da1254b60d7810e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:14.665680 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f738e67d8c2340028da1254b60d7810e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:14.667064 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f738e67d8c2340028da1254b60d7810e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:14.679633 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f738e67d8c2340028da1254b60d7810e'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:14.727711 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1292/2595] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:52:14 2022] DELETE /identity/v3/users/f738e67d8c2340028da1254b60d7810e => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:15.019253 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:15.021831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:15.079496 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:15.082428 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970a3af8-bdb4-4ac1-80e5-602df0d5271d service swift] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:15.083806 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:15.185581 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:15.186509 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:15.186876 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:15.187217 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] PATH_INFO: `/v3/projects/66f8ce6f763241338027d186629b47ce` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:15.188787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] RBAC: Authorizing `identity:delete_project(project_id=66f8ce6f763241338027d186629b47ce)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:15.190874 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:15.239293 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80a64c5-a653-4ff3-a784-7ef118ad08ed admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:15.256038 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66f8ce6f763241338027d186629b47ce', 'request_id': 'req-f8e3680d-89d2-4069-825f-2bc91ba86296'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.263643 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66f8ce6f763241338027d186629b47ce', 'request_id': 'req-f8e3680d-89d2-4069-825f-2bc91ba86296'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.277552 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66f8ce6f763241338027d186629b47ce', 'request_id': 'req-f8e3680d-89d2-4069-825f-2bc91ba86296'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.284365 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:15.295934 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] The token cache is being invalidate because project 66f8ce6f763241338027d186629b47ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81811) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:52:15.296313 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.297769 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.299480 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1304/2596] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:52:15 2022] DELETE /identity/v3/projects/66f8ce6f763241338027d186629b47ce => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:15.343127 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:15.344429 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:15.344938 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:15.345248 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] PATH_INFO: `/v3/projects/8684504b0eff415b9063cebae21ca667` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:15.347779 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8684504b0eff415b9063cebae21ca667)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:15.350109 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:15.423544 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8684504b0eff415b9063cebae21ca667', 'request_id': 'req-a017eada-77fd-4484-a018-be89258a1548'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.433115 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8684504b0eff415b9063cebae21ca667', 'request_id': 'req-a017eada-77fd-4484-a018-be89258a1548'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.440686 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8684504b0eff415b9063cebae21ca667', 'request_id': 'req-a017eada-77fd-4484-a018-be89258a1548'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.467542 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] The token cache is being invalidate because project 8684504b0eff415b9063cebae21ca667 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:52:15.467883 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.468908 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.475583 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1293/2597] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:52:15 2022] DELETE /identity/v3/projects/8684504b0eff415b9063cebae21ca667 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:15.476985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:15.478831 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:15.496053 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e3680d-89d2-4069-825f-2bc91ba86296 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:15.527467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:15.604505 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:15.605326 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:15.605611 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:15.605891 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] PATH_INFO: `/v3/users/e9307ff0301248448ef8b83befda1cf6` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:15.607146 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9307ff0301248448ef8b83befda1cf6)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:15.609129 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:15.677983 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9307ff0301248448ef8b83befda1cf6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.687285 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9307ff0301248448ef8b83befda1cf6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.688060 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9307ff0301248448ef8b83befda1cf6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.688745 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9307ff0301248448ef8b83befda1cf6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.697815 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9307ff0301248448ef8b83befda1cf6'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:15.720612 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1305/2598] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:52:15 2022] DELETE /identity/v3/users/e9307ff0301248448ef8b83befda1cf6 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:15.967974 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:15.970737 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:15.996702 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a017eada-77fd-4484-a018-be89258a1548 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:16.106383 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:16.107467 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:16.107802 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:16.108120 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] PATH_INFO: `/v3/projects/a00f442306cf489f9f7c28839bec1d0e` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:16.109433 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a00f442306cf489f9f7c28839bec1d0e)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:16.111929 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:16.193023 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a00f442306cf489f9f7c28839bec1d0e', 'request_id': 'req-8b53f402-426c-42da-83ad-90708409a0e8'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:16.202174 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a00f442306cf489f9f7c28839bec1d0e', 'request_id': 'req-8b53f402-426c-42da-83ad-90708409a0e8'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:16.209442 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:16.213559 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a00f442306cf489f9f7c28839bec1d0e', 'request_id': 'req-8b53f402-426c-42da-83ad-90708409a0e8'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:16.233524 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] The token cache is being invalidate because project a00f442306cf489f9f7c28839bec1d0e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:52:16.233832 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:16.234923 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:16.238590 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1294/2599] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:52:15 2022] DELETE /identity/v3/projects/a00f442306cf489f9f7c28839bec1d0e => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 00:52:23.468953 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:23.471300 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:23.488787 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18a70e9-a333-4e3f-a39f-1f70704d6b59 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:23.569541 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:23.570340 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:23.570760 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:23.571145 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] PATH_INFO: `/v3/users/531efd120096452caf8c0b7a7f6b9ba1` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:23.573820 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] RBAC: Authorizing `identity:delete_user(user_id=531efd120096452caf8c0b7a7f6b9ba1)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:23.575865 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:23.632174 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:23.634921 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '531efd120096452caf8c0b7a7f6b9ba1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:23.644797 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '531efd120096452caf8c0b7a7f6b9ba1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:23.645925 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '531efd120096452caf8c0b7a7f6b9ba1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:23.646684 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '531efd120096452caf8c0b7a7f6b9ba1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:23.656957 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '531efd120096452caf8c0b7a7f6b9ba1'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:23.680278 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1306/2600] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:52:23 2022] DELETE /identity/v3/users/531efd120096452caf8c0b7a7f6b9ba1 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:23.908620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:23.911085 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:23.938177 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b53f402-426c-42da-83ad-90708409a0e8 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:24.009097 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:24.009747 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:24.010152 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:24.010320 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] PATH_INFO: `/v3/projects/a8a6d900d9d344a694b955f51fa46aa9` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:24.011588 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8a6d900d9d344a694b955f51fa46aa9)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:24.013549 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:24.068275 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8a6d900d9d344a694b955f51fa46aa9', 'request_id': 'req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.076044 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8a6d900d9d344a694b955f51fa46aa9', 'request_id': 'req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.083247 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8a6d900d9d344a694b955f51fa46aa9', 'request_id': 'req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.098841 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] The token cache is being invalidate because project a8a6d900d9d344a694b955f51fa46aa9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:52:24.099276 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.100222 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.103492 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1295/2601] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:52:23 2022] DELETE /identity/v3/projects/a8a6d900d9d344a694b955f51fa46aa9 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:24.107985 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:24.110312 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:24.134119 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8354048-4490-4c94-840d-78ae1fc17f2a admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:24.231408 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:24.232230 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:24.232670 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:24.233137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] PATH_INFO: `/v3/users/25afec59b30d4079a1e9bc73c6433902` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:24.234661 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25afec59b30d4079a1e9bc73c6433902)` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:24.242952 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:24.307694 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25afec59b30d4079a1e9bc73c6433902'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.316507 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25afec59b30d4079a1e9bc73c6433902'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.317467 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25afec59b30d4079a1e9bc73c6433902'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.318223 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25afec59b30d4079a1e9bc73c6433902'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.328551 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.notifications [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25afec59b30d4079a1e9bc73c6433902'} {{(pid=81811) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.359423 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1307/2602] 10.222.0.26 () {58 vars in 1224 bytes} [Wed Oct 19 00:52:24 2022] DELETE /identity/v3/users/25afec59b30d4079a1e9bc73c6433902 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:24.584566 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] Authenticating user token {{(pid=81812) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:52:24.586312 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.fernet_utils [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81812) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:52:24.604620 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ecc8ae-9981-4ad6-8a98-1199e45a08e4 admin admin] Validating token access rules against request {{(pid=81812) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:52:24.681947 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ded9b81a3e894fd88539e86bbeb17ee3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd417ad1948a04e309383739147354388', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81812) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:52:24.682655 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:52:24.682936 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:52:24.683210 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] PATH_INFO: `/v3/projects/c3dc487565414290a5b5fa5c68e1c100` {{(pid=81812) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:52:24.684457 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3dc487565414290a5b5fa5c68e1c100)` {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:52:24.686401 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] RBAC: Authorization granted {{(pid=81812) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:52:24.712464 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:24.748667 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3dc487565414290a5b5fa5c68e1c100', 'request_id': 'req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.755375 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3dc487565414290a5b5fa5c68e1c100', 'request_id': 'req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.763087 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3dc487565414290a5b5fa5c68e1c100', 'request_id': 'req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9'} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.779098 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] The token cache is being invalidate because project c3dc487565414290a5b5fa5c68e1c100 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81812) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 00:52:24.779410 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.780065 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG keystone.notifications [None req-ead742b7-2b9e-4a1b-a93a-ae6719ceb9c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81812) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 00:52:24.781644 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: [pid: 81812|app: 0|req: 1296/2603] 10.222.0.26 () {58 vars in 1233 bytes} [Wed Oct 19 00:52:24 2022] DELETE /identity/v3/projects/c3dc487565414290a5b5fa5c68e1c100 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 00:52:34.352349 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=66,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:52:34.774678 dv-cinder-master-3-iscsi devstack@keystone.service[81812]: DEBUG dbcounter [-] [81812] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81812) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 00:53:47.806822 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] Authenticating user token {{(pid=81811) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 00:53:47.808557 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:53:47.938807 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679526d5-c5f9-4ebb-9e83-307927a2cce1 admin admin] Validating token access rules against request {{(pid=81811) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 00:53:48.050713 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be87a98-5f1b-459e-b105-57311c13158c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '93d2a83a28fd4608994317c2c7b6a723', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fe3331f5cdee4ee6be88cac564cf0233', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81811) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 00:53:48.051598 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be87a98-5f1b-459e-b105-57311c13158c service placement] REQUEST_METHOD: `GET` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 00:53:48.052000 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be87a98-5f1b-459e-b105-57311c13158c service placement] SCRIPT_NAME: `/identity` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 00:53:48.052395 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be87a98-5f1b-459e-b105-57311c13158c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81811) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 00:53:48.055595 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.fernet_utils [None req-0be87a98-5f1b-459e-b105-57311c13158c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81811) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 00:53:48.075790 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be87a98-5f1b-459e-b105-57311c13158c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 00:53:48.078717 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be87a98-5f1b-459e-b105-57311c13158c service placement] RBAC: Authorization granted {{(pid=81811) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 00:53:48.094580 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: [pid: 81811|app: 0|req: 1308/2604] 10.222.0.26 () {60 vars in 1365 bytes} [Wed Oct 19 00:53:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 00:53:58.089137 dv-cinder-master-3-iscsi devstack@keystone.service[81811]: DEBUG dbcounter [-] [81811] Writing DB stats keystone:SELECT=58 {{(pid=81811) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}